A widespread problem in biological research is assessing whether a model adequately describes some real-world data. But even if a model captures the large-scale statistical properties of the data, should we be satisfied with it? We developed a method, inspired by Alan Turing, to assess the effectiveness of model fitting. We first built a self-propelled particle model whose properties (order and cohesion) statistically matched those of real fish schools. We then asked members of the public to play an online game (a modified Turing test) in which they attempted to distinguish between the movements of real fish schools or those generated by the model. Even though the statistical properties of the real data and the model were consistent with each other, the public could still distinguish between the two, highlighting the need for model refinement. Our results demonstrate that we can use ‘citizen science’ to cross-validate and improve model fitting not only in the field of collective behaviour, but also across a broad range of biological systems.