700 milliseconds is roughly the upper bound for a conversation to feel like a conversation. Past that, the pause registers as "thinking"; past a second, it reads as "lag." Most voice agents in production today land around 1.5 to 2.5 seconds end-to-end. Here's how we get under 700ms, what we trade off to do it, and where the budget actually goes.
The latency budget
End-to-end latency, broken down on a real call:
- Audio capture + buffer: ~80ms.
- Speech-to-text streaming: 120-180ms to first partial, ~250ms to final.
- LLM first token: 200-350ms with streaming, longer without.
- Text-to-speech streaming: 80-150ms to first audio chunk.
- Audio playback + network jitter: 50-100ms.
Streaming, end to end
You cannot wait for any stage to finish before starting the next. STT emits partial transcripts every 100ms; the LLM starts generating before the user finishes their sentence; the TTS starts speaking the first clause while the LLM is still generating the last one.
The architectural shape is a pipeline of streams, not a sequence of requests. Each stage has its own back-pressure handling and can interrupt itself when the user starts talking again.
Speculative TTS
The biggest single win in our latency budget came from speculative TTS: synthesizing audio for plausible LLM outputs before the LLM has actually committed to them. We start TTS on the first clause as soon as it ends, even though the LLM might still amend the rest of the sentence.
When the LLM does change course, we cancel the unspoken audio and start a new synthesis from the divergence point. The user almost never hears the seam.
Barge-in
Real-time means the user can talk over the agent. VAD is the floor; on top of that we run a small classifier that distinguishes intentional interruption from filler ("uh-huh," "right," "yeah") in under 120ms. When it fires, we cut TTS playback mid-syllable and restart STT.
Measuring it honestly
If you are not measuring end-to-end latency on real calls in production, you do not actually know what your latency is.
We log five timestamps per turn (user speech end, STT final, LLM first token, TTS first audio, TTS final) and chart the p50, p90, p99 daily. Any regression past p90 wakes someone up.