The voice agent stack, explained: STT, LLM, TTS and telephony
Every voice agent is the same loop: hear the caller, decide what to say, say it, and do it fast enough that the pause feels human. The interesting choices are which layers you buy as a bundle and which you assemble yourself.
1. Transport: phone lines or WebRTC
Phone agents need numbers and SIP trunking from a carrier-grade provider. Browser and app agents usually run over WebRTC. Pick this first — it decides latency budget and which orchestration options fit.
- Twilio — Programmable voice, SIP, and phone numbers widely used for AI phone agents.
- Telnyx — Global SIP trunking, numbers, and realtime communications for voice AI.
- LiveKit — Open-source WebRTC SFU and cloud for realtime audio used by voice agents.
2. Speech-to-text
Streaming STT with good endpointing (knowing when the caller has finished) matters more than raw accuracy benchmarks. Test on your own audio: accents, background noise, phone-quality 8 kHz.
- Deepgram — Streaming speech-to-text API popular for low-latency voice agents.
- AssemblyAI — Speech-to-text and audio understanding APIs with streaming and batch options.
- Speechmatics — Enterprise speech recognition with realtime and batch transcription.
- Gladia — Realtime and batch STT API with multilingual transcription for voice applications.
3. The model
The LLM decides what to say and which tools to call. Speech-to-speech models such as OpenAI Realtime collapse STT, LLM and TTS into one hop, trading control for latency.
- OpenAI Realtime API — OpenAI realtime multimodal API for low-latency speech-to-speech agent experiences.
- Ultravox — Realtime multimodal voice AI platform for low-latency conversational agents.
4. Text-to-speech
Time-to-first-audio and how natural the voice sounds over a phone line are the two things callers notice. Most TTS vendors stream; check voice cloning terms if you need a custom voice.
- ElevenLabs — High-quality text-to-speech and voice cloning used widely in voice agents.
- Cartesia — Low-latency generative TTS models designed for realtime conversational agents.
- Rime — Realtime TTS voices optimized for natural conversational agents.
- PlayHT — AI voice generation and cloning platform with streaming TTS APIs.
5. Orchestration
Something has to manage turn-taking, interruptions (barge-in), tool calls and state. Hosted platforms do this for you; open-source frameworks give you the pipeline to run yourself.
- Vapi — Developer platform for building and deploying voice AI agents with telephony, STT/TTS, and orchestration.
- Retell AI — Voice AI platform for phone agents with low-latency conversation and telephony integrations.
- Pipecat — Open-source Python framework for building realtime voice and multimodal AI agents.
- LiveKit Agents — Framework for building realtime voice AI agents on LiveKit WebRTC infrastructure.
6. Testing and observability
Voice agents fail in ways text chat doesn't — talking over the caller, long silences, mis-heard numbers. Budget for simulated-call testing and per-turn tracing before you scale traffic.
Comparisons
- Vapi vs Retell AI
- Deepgram vs AssemblyAI
- ElevenLabs vs Cartesia
- Twilio vs Telnyx
- Pipecat vs LiveKit Agents
Browse the categories
- Voice Agent Platforms (12)
- Open Source Frameworks (8)
- Speech-to-Text (8)
- Text-to-Speech (9)
- Telephony & SIP (7)
Building one of these tools? Submit it free, or get featured in front of builders reading this.