The Problem

Computer vision models need large amounts of labeled training data to achieve useful accuracy. For most real-world applications, this means someone has to manually draw bounding boxes around thousands of images—a process that's expensive, tedious, and often the biggest bottleneck in building a CV system.

This is especially problematic for ecological monitoring, where the subjects (in our case, fish in stream environments) are difficult to photograph consistently and the available datasets are small. The "cold start" problem: you can't train a model without data, and you can't easily collect data without a working model to guide what to collect.

The Approach

Instead of collecting and labeling thousands of real images, we simulated the target environment using physics-based models. The simulation generates synthetic images with perfect labels already attached—because we placed the objects ourselves, we know exactly where they are.

Specifically, we modeled fish schooling behavior using established physics principles:

• Separation: fish maintain minimum distance from neighbors
• Alignment: fish match heading and speed with nearby individuals
• Cohesion: fish steer toward the average position of their group

These three simple rules produce realistic-looking group movement patterns. By rendering the simulation at various angles, lighting conditions, and water clarity levels, we generated a diverse training set of thousands of images.

Key Results

• 91.8% detection accuracy on real fish images, competitive with models trained entirely on real data
• 90% reduction in the amount of real labeled data needed
• $10,000+ in estimated labeling cost saved
• The approach generalizes: the same pipeline can be adapted for other species and environments

How the Simulation Works

The core simulation uses a Boids-style flocking algorithm. Each fish agent has a position, velocity, and acceleration vector. At each timestep, the three steering forces (separation, alignment, cohesion) are computed based on the agent's local neighborhood, weighted, and applied as acceleration.

The rendering pipeline then captures frames from the simulation and applies randomized environmental variation:

• Water turbidity and color temperature
• Lighting angle and intensity
• Camera position and focal length
• Background substrate (gravel, sand, rock)

Because the simulation controls all these parameters, we can systematically vary them to create a training set that covers the range of conditions the model will encounter in the field.

Training Pipeline

The training process uses a two-stage approach:

1. Pre-training on synthetic data: The model learns basic fish shape, scale, and spatial patterns from thousands of synthetic images.
2. Fine-tuning on real data: A small set of real images (only 10% of what would normally be needed) is used to adapt the model to real-world appearance and artifacts.

This two-stage approach works because the synthetic pre-training gives the model a strong geometric prior—it already knows what fish shapes look like and how they cluster—so the fine-tuning step can focus on texture and lighting differences between simulation and reality.

Implications

The broader point is that physics simulation can serve as a data generation engine for machine learning. When the simulation accurately captures the relevant physics of the target domain, synthetic data can dramatically reduce the cost and time of building CV systems.

This is particularly valuable for ecological monitoring, where data collection is constrained by access, weather, and the behavior of the subjects. It's also applicable to manufacturing inspection, agricultural monitoring, and any domain where real training data is expensive to collect.

Publication

Technologies Used

Python · TensorFlow · OpenCV · NumPy · YOLO · Physics simulation · Synthetic data generation