Version 1: Probability that at least one fish detects a predator at a given distance.
A predator arrives at a certain angle. The probability that a fish sees the predator if there is no occlusion by another fish is p. If N fishes have free view at the angle of the arriving predator, the probability that at leas one fish sees the predator is
1-(1-p)^N. This probability increases with N until it saturates.
If we binnarize the 2D space in 360 angles, and we assume equal probability for the predator arriving each angle, then we have that the probability that at least one fish sees the predator is
, where N_i is the number of fishes with free line of vision at angle i.
The advantage of this cost function is that it takes into account both that many fishes see a certain point (but with the saturation, which I find very reasonable) and that not all fishes look at the same place, leaving others without surveillance.
Since sometimes the shoal will not react until several fish have started evasion, this cost may be generalized to the case when at least n fishes see the predator. However, I think the formula is not simple.
A possible problem with this objective is that it may depend on the binning on angles. My intuition says it will not depend on that (at least not strongly), but at first sight I do not see an easy way of proving it.
Version 2: Distance at which the probability of detecting a predator passes a threshold.
We may assume that the probability that a fish sees the predator (p) depends on distance in a simple way. We may assume that, for long distances (where the probability is much lower than one) the probability grows linearly with the angle subtended by the predator. This angle is inversely proportional to distance, so p=k/d, where k is a constant and d is the distance. Then, for a given angle, the probability that at leas one fish sees the predator is
For a given (constant) value of H, we have that the distance depends on N as
The two versions may not be equivalent, because they will weight differently the benefits of equally distributing the surveillance in all angles. But I would not expect strong differences…