Voice agent architecture: STT, LLM and TTS in real time

Over the past months we have put together several telephone AI agents, and the same three basic questions came up with each of them: how long does it take from the caller finishing a sentence to hearing a reply, what happens when they interrupt, and how well does the system handle Hungarian. In this article we describe the architecture we arrived at and where the pitfalls are.

The pipeline: streaming everywhere

The classic setup has three stages: speech recognition (STT), a language model (LLM), and speech synthesis (TTS). If you run those three one after another, blocking, response time easily goes above 3 to 5 seconds, which is unacceptable on the phone. So we use all three stages in streaming mode:

• the STT produces partial transcripts as the audio arrives, instead of waiting for the end of the sentence;
• the LLM streams token by token, and we pass the text on at the first sentence-ending punctuation mark;
• the TTS synthesises sentence fragment by sentence fragment, and the first audio chunk goes out while the model is still generating the continuation.

The whole thing is an asynchronous pipeline where every stage has its own queueing and its own cancellation signal. Under Node.js this comes naturally: streams, async iterators, AbortController.

Endpointing, turn detection and the latency budget

Most of the delay is not in the models but in the decision: when did the caller finish their sentence? A fixed silence threshold (700 ms, say) is simple but wrong in two ways: with a short pause it cuts the caller off, and with a long threshold the system feels slow.

We use a combined approach: VAD (voice activity detection) on the raw audio, the STT’s own end-of-speech signal, and a simple linguistic heuristic that estimates from the partial transcript whether the sentence is syntactically complete. This is harder in Hungarian than in English, because word order is freer, but question words and suffixes help a lot. A state machine weighs the three signals, and the threshold is adaptive: if the caller speaks slowly, with pauses, the system becomes more patient.

Once endpointing is in order, the budget comes next. What matters for the experience is the time from the caller’s last syllable to the first audio they hear back. We split it roughly like this, with indicative magnitudes:

• endpointing decision: 200-400 ms
• STT finalisation: 100-300 ms
• LLM first token: 300-600 ms
• TTS first audio chunk: 150-300 ms
• network and jitter buffer: 100-200 ms

That adds up to about one to one and a half seconds, which already feels natural. We measure every stage separately (as spans, per call), because if we only see the total we do not know where it degraded. A trick that works well is the filler response: if the LLM’s first token is late, a short, context-dependent acknowledgement (“Right, let me check.”) can already go out, which buys time and keeps the system from seeming mute to the caller.

Media: SIP, WebRTC, RTP and the jitter buffer

The telephony side is the least spectacular yet involves the most work. From the PSTN, RTP arrives over a SIP trunk, typically G.711 (8 kHz, alaw/ulaw) or, if we are lucky, Opus. From the browser, WebRTC arrives with Opus at 48 kHz. STT models, meanwhile, mostly expect 16 kHz PCM, and TTS often produces 24 kHz. So the media layer resamples in every direction, and this is best solved in one place with an unambiguous format contract, otherwise the audio turns “chipmunky” or plays at half speed.

RTP packets do not arrive evenly, so on the receiving side we use a jitter buffer (20-60 ms range, adaptive). On the outgoing side, pacing is required: TTS produces audio faster than the line can carry it, so we send it in 20 ms frames on a clock rather than as it comes. Without that, the jitter buffer on the far side drops packets.

Error handling, barge-in, fallbacks

The caller interrupts: that is barge-in. When it happens, TTS playback has to stop immediately, the outgoing buffer has to be flushed, LLM generation has to be cancelled, and the new user input has to be treated as a new turn. The hard part is handling the truncated response: in the conversation history we store what was actually spoken, not the full generated text.

Timeouts on every external call: if the STT provider does not respond within a given time, we switch to a secondary provider, and for the LLM we fall back to a smaller model. Retry logic is only allowed for idempotent steps; we do not repeat a TTS clip that has already partially played. If everything fails, the agent apologises with a pre-recorded audio file and promises a callback: worse than normal operation, but much better than silence.

Hungarian language specifics

Hungarian is an agglutinative language, so STT models are less certain about rarer word forms than they are in English. What we observed:

• recognition of proper nouns and domain terms (street names, product codes) improves if the STT is given a glossary or a prompt hint;
• numbers, dates and times must not be handed to the TTS raw: instead of “2024. 12. 10.” we pass “kétezer-huszonnégy december tizedike”, complete with the correct suffixes (“tizedikén”, “háromkor”);
• phone numbers and identifiers are read out in groups, with pauses, and then confirmed back;
• Hungarian TTS voices vary widely in prosody, so it is worth comparing several providers on the same set of sentences.

The most important lesson: a voice agent is not an LLM plus two APIs, it is a real-time media system in which the LLM is just one stage. Latency, interruptibility and linguistic normalisation take at least as much engineering work as the prompt.