A client came to us with 380 real defect images for a vision model that needed 5000+ to perform usefully. Buying more meant pausing production lines. Generating them meant diffusion. Here is the pipeline we built, what the model thinks of synthetic vs real data, and how we keep it from going off the rails.
Why diffusion, not GAN or augmentation
Augmentations (rotations, crops, color shifts) preserve the same underlying defect: useful, but limited. GANs are powerful but harder to control. Diffusion with ControlNet lets us generate new defects under controlled conditions: same material, different angles, different severities, different lighting.
The control loop
- Sketch a defect mask manually (or sample from existing masks).
- Feed the mask + a text prompt + the reference material image into ControlNet.
- Generate 8 candidates, score them with a small CLIP-based filter against the prompt.
- Top 3 go into a human review queue.
- Approved samples are labeled, tagged "synthetic," and joined to the training set.
Keeping it honest
We always hold out an "all-real" validation set. The model can train on 90% synthetic data if it wants, but the gold-standard metric is performance on real frames it has never seen.
Every retrain logs the synthetic/real ratio and the all-real validation score side by side. If a higher synthetic ratio starts hurting real-world performance, we throttle generation.
Bias detection
Diffusion models lie in characteristic ways: they over-represent the prompts you used most. Watch your class distribution and rotate prompt templates aggressively.
We hash each prompt and cap how many times it can be used per generation batch. Cheap, very effective.
Results
Across three client deployments: synthetic-augmented training sets reached real-data-only baseline performance with 60% less real data, and exceeded that baseline once the synthetic ratio cleared 4:1.