Barge-in is the difference between a voice agent that feels alive and one that feels like a kiosk. Done well, the agent shuts up the instant the user starts talking, and resumes naturally if the user was just acknowledging it. Done poorly, the agent talks over the user, ignores them, or cuts itself off every time someone clears their throat.
Why plain VAD does not work
Voice Activity Detection just tells you whether sound that looks like speech is happening. It will fire on "uh-huh," on a cough, on the user repeating the last word the agent said as they think. If you cut the agent off on every VAD trigger, every conversation becomes choppy.
The two-stage approach
Stage one is VAD: fast, cheap, and running continuously. When it fires, we open a 150ms window and run stage two, a small classifier that looks at the audio, the partial STT transcript, and the conversational context.
- Backchannel ("yeah," "right," "mm-hm") → ignore, keep speaking.
- Clarification request ("wait, what?") → stop speaking, listen.
- Topic change (new full clause) → stop speaking, listen.
- Non-speech (cough, noise) → ignore.
Staying under 120ms
The classifier has to run faster than the user can finish a syllable. We use a distilled model (about 8M parameters) running on the audio frontend, not a server roundtrip. It sees roughly 250ms of audio plus the last 100ms of STT output.
Per-language tuning
Backchannels vary wildly across languages, and Japanese is famously dense with them. We trained per-language classifier heads on top of a shared encoder. The English head also has dialect-aware tuning, because "right?" as a confirmation and "right" as an interruption are acoustically subtle to tell apart.
Measuring it
The metric that matters: percentage of user interruptions where the agent stops within 200ms, minus percentage of backchannels that incorrectly stop the agent. We chart that daily.