Voice AI: How STT, TTS & AI Calling Agents Work
A practical breakdown of speech-to-text, text-to-speech, realtime speech-to-speech models, telephony, and how to build an AI agent that can actually make and take phone calls.
Voice AI is changing how we interact with software. Instead of typing to an AI, we can simply talk to it, and it can talk back.
But what actually happens behind the scenes when you speak to a voice assistant, and how do these same building blocks let an AI agent make real phone calls? Here's a practical look at the stack.
The basic architecture
A traditional AI voice agent works like this:
- STT (Speech-to-Text): Converts your voice into text.
- LLM: Understands the request and generates a response.
- TTS (Text-to-Speech): Converts the AI's response back into natural speech.
For example, if you say "Schedule an interview tomorrow at 3 PM," STT converts this into text, the LLM understands the request, and TTS speaks the response back to you.
What is speech-to-speech?
Newer realtime AI systems can work directly with audio, skipping the separate text step entirely:
Instead of explicitly managing STT and TTS as separate stages, realtime models can process speech and respond with speech directly, which can make conversations faster and more natural.
Popular tools
STT
- Whisper
- Deepgram
- AssemblyAI
- Google Speech-to-Text
TTS
- ElevenLabs
- Cartesia
- OpenAI
- Deepgram
- Google TTS
- Piper
- Kokoro
Voice agent platforms
- Vapi
- Retell
- LiveKit
- Pipecat
What is telephony?
If you want your AI to make or receive real phone calls, you need telephony infrastructure. Think of it as the bridge between your AI application and the phone network.
Popular providers include Twilio, Telnyx, Vonage, Plivo and Exotel. Protocols such as SIP are also commonly used in professional voice systems.
How to build an AI calling agent
A simple end-to-end architecture looks like this:
For example, a recruitment AI agent could call a candidate, ask screening questions, understand their answers, update the recruitment database and schedule an interview automatically.
The important part is that the LLM can use tools and APIs, which lets the AI actually perform actions instead of just talking.
Open-source projects to explore
If you want to build your own voice AI system, these are great starting points:
- Pipecat — realtime voice and multimodal AI agents
- LiveKit Agents — realtime voice-agent infrastructure
- Whisper — open-source speech recognition
- Piper — lightweight open-source TTS
- Kokoro — open-weight TTS
The path forward
Voice AI is ultimately moving from simple chatbots → conversational agents → AI systems that can actually take actions.
The stack to learn
STT → TTS → LLM → Streaming → WebSockets/WebRTC → Telephony → Tool Calling → AI Agents.
Start with the fundamentals — STT and TTS — then layer in streaming and realtime transport, connect telephony, and finally give your LLM the tools it needs to take real action. That's the difference between a voice chatbot and a true AI calling agent.