Everything so far assumed text arriving in a request and text leaving in a response.
Audio and video break that, and the reason is not the models. It is that speech is a continuous stream with a hard latency budget, while text is a discrete exchange without one.
A person who finishes speaking expects a reply within a few hundred milliseconds. Beyond roughly a second, the pause reads as awkward; beyond two, they assume the system did not hear them and start speaking again. There is no equivalent constraint on a chat interface, where two seconds is fine.
That budget drives the entire architecture.
Two ways to build a voice system
The cascaded pipeline
Convert audio to text, reason in text, convert back to audio.
Diagram, as text
- Microphone leads to Voice activity detection
- Voice activity detection leads to Speech to text
- Speech to text leads to Endpointing
- Endpointing leads to Language model
- Language model leads to Text to speech
- Text to speech leads to Speaker
Each stage is a swappable component with a competitive supplier market, and the intermediate text is inspectable — which matters enormously for debugging, logging and evaluation. This is the production default, and it is what you should propose unless there is a specific reason otherwise.
The cost is latency accumulating across stages, and information lost at each text boundary: tone, emotion, emphasis and interruptions do not survive transcription.
Speech-to-speech
A model that takes audio and emits audio directly, with text as a side channel if at all.
Lower latency and it preserves prosody — the system can hear hesitation and respond to it. The costs are real: fewer providers, less control over intermediate steps, harder evaluation, and a much weaker debugging story when something goes wrong.
Choose it when naturalness is the product. Choose cascaded when reliability, observability and vendor flexibility matter more.
Spending the latency budget
Assume roughly 800ms from the user finishing to hearing a reply. It is spent like this:
| Stage | Typical | Notes |
|---|---|---|
| Endpointing | 200–500ms | Deciding they have finished — often the largest slice |
| Speech to text | ~100ms | Streaming, mostly done as they speak |
| Model first token | 200–500ms | Time to first token, not full response |
| Text to speech first audio | ~100ms | Streaming |
Three design consequences follow directly, and they are the substance of this lesson.
Stream everything. Nothing waits for the previous stage to complete. Transcription runs while the user speaks; the model starts generating on partial input where possible; speech synthesis begins on the first sentence rather than the full answer. Only the first audio chunk is on the critical path — the rest is produced while the user listens.
Endpointing is the hard part. Detecting that someone has finished rather than pausing mid-thought is genuinely difficult. Wait too long and the system feels slow; too little and you interrupt. Silence thresholds are the naive approach; better systems use prosodic cues and semantic completeness — "I'd like to book a" is clearly unfinished regardless of the pause.
Time to first token matters more than total. The serving lesson's distinction becomes a product requirement: once audio starts playing, the rest can generate while the user listens. Optimise the first token; total generation time is far less visible.
Concurrent streams
A voice or video system runs several things at once: audio capture, transcription, model generation, tool execution, synthesis and playback — all live simultaneously, not in sequence.
Modelling this as a request-response flow does not work. It is an event-driven system, and the design questions are ordinary concurrency questions:
- What happens to in-flight generation when the user interrupts?
- If a tool call takes three seconds, what does the user hear meanwhile?
- If transcription revises an earlier word after the model has started, what then?
Treat each stage as a component consuming and emitting events, with explicit state for the conversation. The failure mode is race conditions producing incoherent behaviour — replying to a sentence the user retracted, or two responses playing over each other.
Images and video
Simpler, because there is no real-time constraint in most applications.
Images are the straightforward case: send with the prompt, get text back. Design considerations are resolution against cost, since images consume substantial token budget, and whether you need the whole image or a region.
Documents are frequently better handled as images than as extracted text — layout, tables and figures carry meaning that text extraction destroys. A scanned invoice often works better sent visually than run through OCR.
Video is the expensive case, and the usual approach is to avoid processing it as video: sample frames at intervals, process the audio track separately, and combine. Full-frame-rate understanding is rarely necessary and rarely affordable.
Multimodal embeddings allow searching images with text queries and vice versa, since both map into a shared space — the embeddings lesson generalised beyond text.
Cost and evaluation
Cost is dominated by input volume. Images and audio consume far more budget per second of content than text. Downscale images to the smallest resolution that preserves what matters, and sample video rather than processing continuously.
Evaluation needs new dimensions. Beyond answer quality: transcription accuracy on your actual audio — accents, background noise, domain vocabulary — endpointing precision, interruption handling, and end-to-end latency at the tail. A system with a good median and a bad P95 feels unreliable, because the occasional long pause is what users remember.
Evaluate on realistic audio, not clean studio recordings. Real calls have noise, overlap and accents, and a pipeline tuned on clean input degrades sharply on them.
Check your understanding
Common mistakes
Building stage-by-stage instead of streaming. Latency accumulates and the system feels sluggish.
Naive silence-threshold endpointing. Either interrupts or feels slow; there is no good fixed value.
No interruption handling. The single most damaging omission in a voice product.
Optimising total latency. First audio is what users perceive.
Testing on clean audio. Real input has noise, accents and overlap.
Full-resolution images by default. Large, avoidable cost.
Processing video frame by frame. Sample instead.
Conversation state that ignores what was actually heard. Produces incoherent follow-ups after an interruption.
Practice
You have an 800 ms budget from the user finishing speaking to hearing the first audio. Allocate it across the pipeline and say what you would do to fit.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Work out what must be serial and what can overlap.
Show a strong answer
A workable allocation:
- Endpoint detection — 150 ms. The delay after silence before deciding the turn is over. This is a deliberate wait, and it is the largest single lever: cutting it to 50 ms saves 100 ms and causes interruptions.
- Final transcription — 50 ms. Only the tail, because transcription should have been running incrementally throughout the utterance rather than starting now.
- Model time to first token — 400 ms. The largest genuine cost.
- Speech synthesis of the first chunk — 150 ms.
- Network and buffering — 50 ms.
What makes it fit:
- Transcribe while they speak, not after. By the time they stop, all but the last words are already text. Treating transcription as a stage that begins at the end wastes the whole utterance's duration.
- Start the model on a partial transcript where the intent is already clear, and cancel if the ending changes it. Speculative and effective for common phrasings.
- Stream into synthesis at the first sentence boundary, not at the end of generation. This is what converts "wait for the whole answer" into "start speaking almost immediately".
- Keep the first sentence short by design. Prompt for an opening that begins with a brief acknowledgement, so synthesis has something to say while the substantive part is still generating.
- Prefill-cache the system prompt so time to first token is spent on the actual request.
What I would sacrifice if it still does not fit: model size before endpointing patience. A slightly weaker answer is less noticeable than being interrupted mid-sentence — and an assistant that talks over you is the failure users describe as "it doesn't listen".
Compare a cascaded pipeline (speech-to-text, model, text-to-speech) with an end-to-end speech model. When would you choose each?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask what each stage boundary gives you and what it costs.
Show a strong answer
Cascaded — three components with text between them.
- For: every stage is inspectable and independently replaceable. You can log the transcript, evaluate the language model with text-based evaluation sets, apply text guardrails, and swap the synthesis voice without touching anything else. Debugging is tractable because you can see exactly what the model was asked.
- Against: latency accumulates across stages, and information is lost at each boundary. Tone, emphasis, hesitation and emotion are discarded by transcription and cannot be recovered — so the model cannot tell an uncertain "yes?" from a confident "yes".
End-to-end — audio in, audio out.
- For: lower latency with no stage handoffs, and it can use prosody. It can also produce prosody appropriate to the content, which cascaded synthesis approximates from text alone.
- Against: far harder to inspect. There is no transcript to log unless one is produced separately, text guardrails do not apply directly, and evaluation is harder because the output is audio. Provider choice is narrower and you cannot mix best-of-breed components.
When I would choose each:
- Cascaded for anything with compliance, auditing or complex tool use. A support assistant that must log what was said, apply content policy and call internal systems needs the text boundary — the observability is not optional.
- End-to-end for conversational, low-stakes, latency-critical interactions where naturalness is the product: a companion, a language-practice partner, a hands-free assistant.
The hybrid worth naming: end-to-end for the conversational path, with a parallel transcription stream purely for logging and guardrails. It costs an extra transcription and recovers most of the observability, which is often the right trade.
Key takeaways
- Voice is hard because of a hard latency budget — beyond about a second the pause reads as failure.
- Cascaded pipelines are the production default: swappable components, inspectable intermediate text, easier evaluation. Speech-to-speech is lower latency and preserves prosody at the cost of observability.
- Stream every stage so only the first audio chunk is on the critical path.
- Endpointing is the hardest component and often the largest share of the budget; semantic completeness beats silence thresholds.
- Interruption handling is a requirement, and it includes correcting conversation state to what was actually heard.
- Multimodal systems are event-driven with concurrent streams, not request-response.
- Images cost more than text — downscale; sample video rather than processing every frame.
- Evaluate on realistic audio and watch tail latency, since occasional long pauses define perceived reliability.
Further reading
- Alec Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (ICML 2023) — the Whisper paper.
- Jonathan Shen et al., "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (ICASSP 2018).
- Zalán Borsos et al., "AudioLM: a Language Modeling Approach to Audio Generation" (IEEE TASLP, 2023).
- Alec Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021) — CLIP, for the image side.
- Nicholas Gibbs and Matthew Aylett, "Turn-taking in spoken dialogue systems" (Interspeech tutorials) — endpointing and its trade-offs.