How Voice AI Works: Audio, Models, Turn-Taking and Latency

A voice AI agent must do more than convert speech into a spoken answer. It has to transport live audio, detect when a person starts and stops talking, understand the request, call any required business tools, generate a concise response, play audio, and handle interruptions. All of this must happen quickly enough that the exchange still feels like a conversation.

Two architectures are common: a chained pipeline that explicitly connects speech-to-text, an agent or language model, and text-to-speech; and a speech-to-speech system that processes live audio in one realtime session.

The complete voice-agent loop

1. Capture and transport audio: a phone network, WebRTC connection, or app streams the user's audio to the service.
2. Detect the turn: voice activity detection or another endpointing method decides when speech has started and whether the user is finished.
3. Interpret the request: the system transcribes speech or reasons directly over audio while maintaining conversation state.
4. Use tools when needed: it may check a calendar, retrieve an account, or route the call—under explicit permission and validation rules.
5. Create the response: the agent produces a short answer suited to spoken delivery.
6. Synthesize and stream audio: speech begins playing as soon as a useful first segment is ready.
7. Handle interruption: if the user speaks, the system stops or truncates playback and listens to the new turn.

Leaving out transport, turn detection, and tool behavior makes the familiar “STT ? LLM ? TTS” diagram incomplete.

Chained voice architecture

A chained pipeline keeps the three core transformations visible:

Live audio ? speech-to-text (STT) ? agent/LLM and tools ? text-to-speech (TTS) ? live audio

1. Speech-to-text: turn audio into usable text

Streaming STT produces interim and final transcripts while audio is arriving. It must cope with call compression, accents, background noise, names, domain terminology, numbers, and people correcting themselves mid-sentence.

Recognition quality depends on the actual audio and vocabulary, not a provider's demo alone. Test with representative calls and evaluate:

• word error rate and, more importantly, errors in names, dates, amounts, and identifiers;
• how quickly useful interim text appears;
• how the service marks a final utterance;
• performance in noise, crosstalk, and the languages your users speak;
• whether vocabulary hints or model adaptation are available.

Google documents streaming recognition as a bidirectional stream that returns results while audio is processed. Other STT services expose comparable streaming concepts, but their events and tuning controls differ.

2. Agent and language model: decide what to say or do

The agent receives a transcript and conversation context, identifies the task, decides whether a tool is required, and drafts the response. For “Move my appointment to Tuesday,” it may need to verify identity, retrieve the existing appointment, query availability, ask a clarifying question, and only then propose a change.

The model should not be the sole source of business truth. Appointment slots, balances, eligibility, policies, and order status belong in authoritative systems reached through validated tools. High-impact actions need confirmation and an audit trail.

3. Text-to-speech: turn the response into audio

TTS converts response text into playable speech. Voice quality involves intelligibility, pronunciation, pacing, consistency, and suitable prosody. The fastest way to improve spoken output is often to improve the text: short sentences, one question at a time, and no dense lists or markup.

Test names, acronyms, numbers, addresses, and mixed-language phrases. Choose a voice that remains clear over the actual phone or device channel, not only through studio headphones.

Speech-to-speech architecture

In a speech-to-speech design, a realtime model accepts audio and produces audio within a live session. OpenAI's current voice-agent guidance presents this as a strong fit for natural, low-latency conversation, turn-taking, interruption, and realtime tool use.

The architecture is simpler at a high level:

Live audio ? realtime audio model and tools ? live audio

This does not mean there are no internal recognition or synthesis processes. It means the application does not have to orchestrate separate STT, text-agent, and TTS services for every turn.

Architecture	Strengths	Trade-offs	Good fit
Chained STT ? agent ? TTS	Explicit transcripts, interchangeable components, detailed text controls, easier stage-by-stage inspection	More services and handoffs to tune; text can discard some vocal information	Predictable workflows, transcript-heavy support, existing text-agent systems, approval steps
Speech-to-speech	Natural turn-taking, direct audio context, fewer application-level conversions	Provider-specific behavior; audio sessions and debugging require different observability	Fluid conversation, barge-in, low first-audio delay, realtime assistance

Choose based on control, auditability, quality, cost, integration needs, and tested user experience—not on the assumption that one architecture is always faster or more accurate.

Turn detection is part of the intelligence

A voice agent must know whether a pause means “I am done” or “I am thinking.” If endpointing is too aggressive, it interrupts the user. If it waits too long, every answer feels sluggish.

Common approaches include:

• Voice activity detection (VAD): detects speech and silence from the audio signal.
• Semantic turn detection: considers whether the utterance sounds complete, not only how long the silence lasts.
• Push-to-talk: gives the user explicit control and can simplify noisy environments.
• Manual commit: the application decides when to send the accumulated audio for a response.

OpenAI documents VAD events for speech start and stop in Realtime sessions. Deepgram similarly documents endpointing based on speech and silence. The correct threshold must be tested with your callers, languages, and call conditions.

Why voice AI feels slow

End-to-end delay includes more than model inference:

Delay source	What contributes	What to measure
User turn completion	Silence threshold, semantic endpointing, network jitter	End of actual speech to committed turn
Recognition	Audio buffering, STT processing, finalization	Committed turn to usable transcript
Reasoning and tools	Model response, database or API calls, retries	Transcript or turn commit to first response token or tool result
Speech generation	Text chunking, TTS processing, audio buffering	Text available to first playable audio
Transport and playback	Telephony, WebRTC, codecs, client buffering	Audio generated to audio heard

Measure the whole path with timestamps from the user's last spoken audio to the first audible response. Then break it into stages. A vendor's isolated model figure does not predict the experience across your network, tools, and playback client.

Streaming reduces perceived delay

Without streaming, an application may wait for the entire transcript, complete response, and full audio file. With streaming, interim recognition can begin during the user's speech, the model can emit the response incrementally, and TTS can play the first phrase while later text is still being generated.

Streaming introduces obligations:

• do not speak a claim before required data or approval is available;
• buffer enough text to avoid broken pronunciation and unstable sentence starts;
• cancel model and TTS work when the caller interrupts;
• avoid paying for or logging abandoned work unnecessarily;
• keep the spoken response consistent with any later transcript.

Barge-in and interruption handling

Natural callers interrupt. When speech begins during playback, the system should stop outgoing audio, record how much the user actually heard, and process the new turn. If the conversation history incorrectly contains the entire unplayed response, the agent may later assume it told the user something it never delivered.

OpenAI's Realtime conversation documentation describes truncating the model response at the interruption point. Other platforms expose different events, so verify cancellation behavior end to end.

A realistic appointment example

1. The caller says, “I'd like to move my appointment to next Tuesday.”
2. Turn detection waits long enough to avoid cutting off the sentence.
3. The system extracts the intent and identifies which appointment is meant.
4. A verified tool retrieves the existing booking and Tuesday availability.
5. The agent says, “I found openings at 10 a.m. and 2 p.m. Which do you prefer?”
6. If the caller interrupts after “10 a.m.,” playback stops and the new speech becomes the next turn.
7. The agent repeats the chosen date and time and asks for confirmation before committing the change.

The last confirmation matters more than saving a fraction of a second.

What to test before real calls

• noisy rooms, speakerphone, mobile networks, and low-quality telephony audio;
• accents, languages, code-switching, names, dates, amounts, and account numbers;
• long pauses, false starts, interruptions, and two people speaking;
• tool timeouts, empty results, duplicate requests, and downstream errors;
• prompt injection or malicious text returned by tools;
• privacy notices, consent for recording or transcription, retention, and redaction;
• escalation to a human and a clear way to stop the automation.

For product comparisons, TipsMake's AI voice generator guide focuses on voice creation, while its overview of AI agent-building platforms covers broader workflow tools.

A good voice agent is not simply three APIs connected in sequence. It is a measured realtime system whose turn-taking, tools, spoken language, interruption behavior, and safety controls all work together.