How Voice AI works

Understand the 3-component Voice AI architecture – Speech-to-Text, LLM, and Text-to-Speech – and how latency affects the caller experience.

 

Think about your most recent pleasant phone conversation. The other person listened, understood you, and responded naturally. There were no awkward silences. No "excuse me, could you repeat that?" after every sentence.

That's what a Voice AI agent needs to do—in less than a second. And it accomplishes it using three components that work together in a process. Understanding this process is key to building agents that feel natural rather than robotic.

3 main components

All Voice AI agents—whether built on Vapi, Retell, Bland, or custom code—operate on the same underlying architecture:

Caller says → [Sequence Number] → [Full Name] → [Sequence Number] → Caller listens for response

Let's analyze each component.

1. Item No.: Ears (Speech-to-text conversion)

STT listens to the caller's audio and converts it into text. That's all. But doing it well is harder than you think.

The STT tool needs to process:

1. Accent and dialect. Callers from Texas sound different from callers from Boston.
2. The surrounding noise. Cars, cafes, wind, other people talking.
3. Specialized terminology. Medical terms, legal language, product names created by your company.
4. Interruption. People didn't wait for the AI ​​to finish speaking before interrupting.

Popular STT providers include: Deepgram (fast, good accuracy), Google Cloud Speech, Whisper (an OpenAI model, also runs locally), and AssemblyAI.

 

Latency: 100-500ms. Deepgram and Google are generally faster. Whisper is accurate but slower unless you run it on powerful hardware.

✅ Quick check : What else does the STT tool need to process besides basic speech recognition?

Answer : Tone of voice, background noise, technical jargon, and interruptions.

2. LLM: The Brain (Big Language Model)

After the caller's speech is converted into text, the LLM will figure out its meaning and generate a response. This is where "intelligence" comes in.

LLM processing:

1. Understand the intent. "I need to reschedule" means the caller already has an appointment and wants to change it.
2. Maintain context. Remember that the caller said their name was Sarah and their appointment was on Thursday.
3. Make a decision. Should the employee schedule an appointment, forward the call to the actual person, or ask further clarifying questions?
4. Create natural-sounding language. Create text that sounds like what a real person would say.

Popular LLM providers for speech include: GPT-4o and GPT-4o-mini (OpenAI), Claude (Anthropic), and Gemini (Google). For speech specifically, faster models like the GPT-4o-mini are often preferred because latency is more important than maximum intelligence.

Latency: 200-2000ms. This is often the biggest bottleneck. A complex response from a large model can take up to 2 seconds. A simple acknowledgment from a fast model can take 200ms.

3. TTS: Speech (Text-to-speech)

TTS takes LLM's text feedback and converts it into audio. Modern TTS doesn't sound like a robot reading a script—it sounds like a person speaking.

TTS processing tool:

1. Intonation. The rise and fall of your voice. Questions should sound like questions.
2. Rhythm. Natural pauses between phrases and sentences.
3. Emotions. Warmth for a greeting, empathy for a complaint, enthusiasm for good news.
4. Choose your voice. Male, female, young, old, formal, informal - you can select.

Popular TTS providers include: ElevenLabs (best quality, most natural), PlayHT, Cartesia (ultra-low latency), and cloud service providers (Google, Amazon Polly, Azure).

 

Latency: 200-800ms. ElevenLabs delivers excellent sound but increases latency. Cartesia is faster but the output sound is slightly less natural.

✅ Quick check : Which component in the processing workflow typically causes the most latency? Answer : LLM, at 200-2000ms.

Why is speed important?

This is where everything comes together. Add up the latency from all three components and you'll have the total response time:

Ingredient	Fast	Typical	Slow
No.	100ms	250ms	500ms
LLM	200ms	500ms	2000ms
TTS	200ms	400ms	800ms
Total	500ms	1150ms	3300ms

How will the caller feel?

1. Below 500ms: Instantaneous feeling. Like talking to a sharp-witted person.
2. 500-800ms: Natural. Most people won't notice the difference. This is the ideal range.
3. 800-1500ms: Acceptable. There's a slight pause, but it still sounds natural, like a conversation.
4. 1500-2500ms: Anxious. The caller begins to wonder if the line has been cut.
5. Above 2500ms: Bad. People hang up or start repeating themselves.

Retell, one of the leading platforms, achieves end-to-end latency of around 600ms. That's the standard to aim for.

Stacked architecture versus Speech-to-Speech (S2S)

Everything we've discussed so far has been cascading architecture – sound going through three distinct steps in sequence. But there's a newer approach.

Speech-to-text conversion (Most common method)

Audio → Serial Number → Text → Letter of Reference → Text → Text → Text → Audio

Advantage

1. Flexible.
2. It is possible to combine multiple different suppliers.
3. Replace Deepgram with Whisper, or GPT-4o with Claude.
4. Each component is independent.

Disadvantages

1. Three conversion steps mean three times the delay. And information is lost during the conversion process – tone, emphasis, emotion are not preserved after speech-to-text conversion.

Speech-to-speech (S2S)

Audio → S2S Model → Audio

Speech-to-speech models process audio directly. There is no text conversion in between. OpenAI's real-time API uses this approach.

Advantage

1. Lower latency (skipping two conversion steps).
2. Maintain your vocal tone.
3. It is possible to detect emotions in the caller's voice.

Disadvantages

1. Fewer options - this is newer technology.
2. Less control over individual components.
3. Debugging is more difficult because there is no text log in between.

Which one should you use? For most business use cases in 2026, cascading architecture remains the practical choice. Many platforms support it, it's easier to debug, and the tools are more complete. But keep an eye on S2S – that's the trend heading.

 

✅ Quick Check : What are the main trade-offs between cascading architecture and S2S?

Answer : Cascading offers greater flexibility and easier debugging but higher latency. S2S offers lower latency and richer audio processing but fewer options and is more difficult to debug.

Stream: Don't wait for a full answer

Here's a technique that helps voice agents work significantly faster: Streaming.

Without a stream, the process would wait for the LLM to generate the entire response before the TTS begins converting it to audio. With a stream, the LLM sends text snippets as they are generated, and the TTS begins speaking the first sentence while the LLM is still writing the second.

Không sử dụng stream: LLM tạo ra phản hồi đầy đủ ————————→ TTS chuyển đổi toàn bộ văn bản → Âm thanh phát Với stream: LLM tạo ra đoạn 1 → TTS đọc đoạn 1 → Người gọi nghe thấy LLM tạo ra đoạn 2 → TTS đọc đoạn 2 → Người gọi nghe thấy LLM tạo ra đoạn 3 → TTS đọc đoạn 3 → Người gọi nghe thấy

This can reduce perceived latency by 500ms or more. All major Voice AI platforms support streaming, and you should always keep this feature enabled.

In summary: A real call

Follow a single exchange throughout the entire process:

1. The caller said, "Hello, I'd like to schedule an appointment for next Tuesday."
2. STT (150ms): Convert audio to text: "Hello, I would like to schedule an appointment for next Tuesday."
3. LLM (400ms): Understand intent (scheduling request), check the system for available time slots, generate a response: "Absolutely! I have available slots at 10 AM and 2 PM next Tuesday. Which time would suit you better?"
4. TTS (300ms): Convert the answer into a natural-sounding audio with a rising intonation at the question.
5. Total: 850ms. The caller barely noticed the pause.

And with the streaming feature, the caller hears "Sure! I have time available at." while the LLM is still completing the sentence. Perceived latency is reduced to around 500ms.

It's AI voice. Three components, one process, under a second.

Key points to remember

1. Voice AI uses a three-stage process: STT (ear) → LLM (brain) → TTS (voice)
2. Total latency is the sum of all three phases - aim for below 800ms for a natural experience.
3. LLM is the biggest bottleneck in terms of latency (200-2000ms), so faster models are crucial.
4. Stacked architecture offers flexibility; S2S offers lower latency but fewer options.
5. Stream allows callers to hear feedback before a full response is generated, significantly reducing the waiting time for feedback.

1. Question 1:

   Why is streaming important in Voice AI?

   1. A. Use less bandwidth
   2. B. Improve sound quality
   3. C. Required by telecommunications regulations - a common misunderstanding leading to suboptimal results.
   4. D. Allows users to begin listening to feedback before a complete answer is generated.

   EXPLAIN:

   Stream sends audio segments as they are generated, so callers begin hearing the response almost immediately—instead of having to wait for the entire response to be fully generated first.

2. Question 2:

   What are the main advantages of a Speech-to-Speech (S2S) architecture compared to a cascading architecture?

   1. A. Lower operating costs
   2. B. Support for more languages
   3. C. Eliminate text conversion steps, reduce latency.
   4. D. Producing higher-quality sound – an understandable assumption that experienced individuals will quickly question.

   EXPLAIN:

   Speech-to-speech models completely bypass the STT and TTS steps, processing the audio directly. This eliminates the two conversion steps and can significantly reduce latency.

3. Question 3:

   What is the typical overall latency for a well-optimized Voice AI response?

   1. A. Approximately 600ms
   2. B. Under 100ms
   3. C. Approximately 3 seconds
   4. D. 5 - 10 seconds

   EXPLAIN:

   Well-optimized systems like Retell achieve an overall latency of around 600ms, creating a natural feel in conversation. Below 500ms feels instantaneous, while anything over 2 seconds feels awkward.

4. Question 4:

   In the voice AI pipeline, what is the function of the STT (Speech-to-Text) component?

   1. A. Creating speech sounds from text
   2. B. Processing meaning and generating feedback
   3. C. Converting speech to text
   4. D. Route calls to the appropriate department.

   EXPLAIN:

   STT (Speech-to-Text) is the 'ear' of the system – it listens to what the caller says and converts the audio into text that LLM (Logical Learning Management) can process.

Submit your work

 

Training results

You have completed 0 questions.

-- / --

Review the lesson