Word Error Rate is a useful metric for transcription. It is close to useless as a metric for whether a voice agent is good. A bot can have 2% WER and be infuriating to use; another can have 8% WER and feel great. The signal lives elsewhere. Here is the rubric we evolved.
The four dimensions we score
- Turn-taking: did the agent pause when interrupted? Did it wait for the user to finish? Was its response timed naturally?
- Recovery: when the user said something unexpected, did the agent ask a useful clarifying question, or did it confidently do the wrong thing?
- Task completion: did the actual job get done, end to end?
- Trust signals: did the agent say things that turned out to be wrong? Did it admit uncertainty when it should have?
The harness
We generate synthetic conversations from a script grammar: openings, escalations, edge cases, distractions. Each scenario plays through with a known ground truth for what should happen. We replay them nightly against the production agent and any candidate version.
The synthetic user is also a model, prompted with personality, accent, and behaviors (mid-sentence corrections, filler, topic drift). It is not perfect, but it surfaces regressions a static script never would.
Grading
A judge model rates each dimension on a 1-5 scale with a structured rubric. We sample 20% of judgments for human spot-checking weekly; agreement runs around 0.85. The model judge is fine for trend detection; the human reviewers catch the systematic biases.
When we know we have regressed
Any dimension drops more than 0.3 points on the rolling weekly average → block the deploy and triage.
What this caught that WER missed
A model upgrade that lowered WER by half a point also tanked turn-taking by 0.6: the new model sounded more "polished" but interrupted users far more often. We'd have shipped it blind.