ViTs are not strictly better than CNNs, and CNNs are not the safe boring choice anymore either. The right architecture depends on three boring questions: how much data do you have, what is your inference budget, and can you fine-tune. Here is how we route a new project.
Dataset size is the first filter
ViTs are data-hungry. Below ~50k labeled images, a pretrained CNN almost always wins out of the box. Above that, ViTs start to pull ahead. Above ~500k, the gap is significant. Below 10k, a CNN backbone with strong augmentation will beat a ViT every time, even at significant cost in compute.
Latency budget
CNN inference scales nicely on edge devices: convolutions are friendly to NPUs, GPUs, and quantization. ViTs are harder to deploy on constrained hardware; their attention layers don't love INT8 as much as conv layers do. If your target is sub-20ms on a Jetson, lean CNN.
Fine-tuning feasibility
ViT fine-tuning needs more memory and more careful learning-rate scheduling. If your team hasn't done it before, the first project takes longer than expected. CNNs are forgiving: swap the head onto a frozen backbone, and you usually get something workable.
The hybrid case
Frequently the right answer is "both." A small CNN backbone extracts features cheaply, a transformer head on top reasons about spatial relationships. We use this for high-resolution defect detection where global context matters but a pure ViT would blow the latency budget.
The decision tree, condensed
- <10k labels? CNN. No question.
- 10k-50k labels, edge deployment? CNN.
- 10k-50k labels, server? Try both; CNN is the safer default.
- >50k labels, server, accuracy-critical? ViT.
- Anything where you need global context and tight latency? Hybrid.