Facts at a glance
- Stack first – SIP/CPaaS, recording, identity, and observability sit under the speech model, not after it.
- VoIP-first rails – FCC (June 30, 2025; report May 2026): ~44.0M business interconnected VoIP (+4.1% YoY), ~83.6% of U.S. business fixed voice.
- UCaaS hosts adjacency – Metrigy: UCaaS $23.0B in 2025 (+6.1%); ~58.6% of businesses use UCaaS alone; Big 4 ~53% of seats.
- Hybrid endpoints – Gallup: ~52% hybrid / 26% remote / 22% on-site among remote-capable workers.
- Design latency budgets – Prove WebSocket/SIP paths under load before you scale agents.
AI voice infrastructure is the production stack under the speech model: SIP or CPaaS connectivity, UCaaS adjacency, recording and compliance, identity, and observability—not a model demo bolted onto an unmonitored trunk.
That stack rides a VoIP-first market. The FCC’s Voice Telephone Services report as of June 30, 2025 (released May 8, 2026) puts U.S. business interconnected VoIP near 44.0 million subscriptions (+4.1% YoY)—about 83.6% of business fixed voice—while OTT business VoIP grew +9.3% YoY (FCC Voice Telephone Services; VoIP statistics). Across all fixed voice, interconnected VoIP reached about 63.4 million subscriptions versus roughly 15.0 million switched lines. Metrigy reports global UCaaS at $23.0 billion in 2025 (+6.1%), with about 58.6% of businesses on UCaaS alone, worldwide seats near 117.1 million (+6.9%), and Big-4 vendors near ~53% of seats (Microsoft ~22.2%, Cisco ~16.5%, Zoom ~8.3%, RingCentral ~6.4%) (Metrigy). Hybrid work keeps softphones in the perimeter: Gallup finds ~52% hybrid / 26% fully remote / 22% fully on-site among remote-capable U.S. workers (Gallup).
Use this guide to design latency, failover, and compliance first, then shortlist building blocks in our provider comparison hub.
Latency Budget
A human-to-human phone call introduces roughly 150–250 ms of mouth-to-ear latency before the listener perceives a conversational lag. Voice AI systems in 2026 operate under the same constraint, but the budget must be split across more stages. Breaking down a representative production pipeline running Deepgram for ASR, GPT-4o for inference, and ElevenLabs for TTS, streamed over LiveKit’s WebRTC infrastructure:
| Stage | Typical Latency | Notes |
|---|---|---|
| Audio capture + VAD | 20–60 ms | Silero-VAD on server CPU dominates; WebRTC-native VAD is less accurate |
| STT (streaming ASR) | 150–400 ms | Deepgram Nova-2 streaming at ~300 ms vendor-stated; NVIDIA Riva on A100 achieves ~80–120 ms for English |
| LLM inference (first token) | 300–800 ms | GPT-4o Realtime at ~400–600 ms; open-weight models on Groq at ~200–400 ms |
| TTS generation (first audio) | 100–600 ms | ElevenLabs Turbo v2.5 at ~150–300 ms vendor-stated; NVIDIA Riva FastPitch+HiFi-GAN at ~50–100 ms on A100 |
| Network transport | 50–300 ms | WebRTC Opus at ~20–50 ms edge-to-edge; WebSocket PCM adds ~50–100 ms encoding overhead; PSTN/SIP adds 100–300 ms carrier routing |
End-to-end, a well-tuned voice AI system in 2026 delivers 800–1,500 ms of total latency from the moment a user stops speaking to the moment they hear the first syllable of the response. Systems running speech-to-speech models that bypass the STT-LLM-TTS chain entirely — such as OpenAI’s GPT-4o Realtime audio mode — can cut this to 400–800 ms by collapsing transcription and generation into a single multimodal pass.
The hard truth is that turn-detection logic, not model inference, is the dominant driver of latency in 2026. Voice activity detection (VAD) must decide when the user has finished speaking — too aggressive and the AI interrupts mid-sentence; too conservative and the user waits in silence. Deepgram, ElevenLabs, and Vapi each ship their own turn-detection heuristics, layered on top of Silero-VAD or proprietary models, and the configuration trade-offs between responsiveness and interruption avoidance remain the single most impactful tuning decision in production voice AI deployments.
Two Competing Voice Loops
The industry has split into two architectural camps for the voice loop itself. The STT → LLM → TTS pipeline, championed by Deepgram, ElevenLabs, and NVIDIA Riva, treats each stage as a separable, swappable microservice. It gives teams the flexibility to swap Whisper for Deepgram for Riva, or ElevenLabs for Play.ht, without touching the rest of the stack. The trade-off is serial latency: each stage must complete before the next can begin, and transcription inherently discards prosody, pitch, and emotional valence that the LLM could otherwise reason about.
The speech-to-speech model, deployed by OpenAI’s GPT-4o Realtime and Google’s Gemini Live, feeds encoded audio directly into a multimodal model that generates audio tokens in return. This collapses the three-stage pipeline into one model call, preserving prosodic information and cutting end-to-end latency by 40–60%. The downside is vendor lock-in: the STT, LLM, and TTS are inseparable, and the model is a black box. Teams using speech-to-speech cannot independently upgrade any single stage.
Vapi, the voice AI platform that abstracts away the infrastructure layer entirely, supports both pipelines and routes developers to the speech-to-speech path when latency budgets are tight and the use case (customer support, sales outreach) tolerates the opacity.
Where the Transport Layer Fits
Beneath both architectures runs the transport layer: the protocol that moves audio between the user’s device and the agent’s inference backend. In 2026, this is almost always WebRTC for browser and mobile clients, and WebSocket for backend-to-backend links. The distinction matters because the transport layer determines bandwidth consumption, codec negotiation, NAT traversal behavior, and whether a conversation can scale from a 1:1 human-agent call to a multi-party conference with multiple human participants and a server-side agent.
WebSocket Architectures: The Backend-to-Backend Workhorse
WebSocket dominates the backend-to-backend audio path in 2026 for reasons that are pragmatic rather than architectural. When Deepgram receives a streaming ASR request from an application server, or when ElevenLabs streams TTS audio back to a Vapi agent orchestrator, the transport is almost always a WebSocket connection carrying uncompressed PCM audio encoded as base64 inside JSON frames.
Why WebSocket, Not WebRTC, Between Servers?
WebRTC was designed for peer-to-peer media over unreliable networks. Its signaling layer (Session Description Protocol, or SDP), its ICE candidate negotiation, and its DTLS handshake introduce hundreds of milliseconds of setup overhead on every new session. Between two servers in the same AWS region or across peered VPCs, that overhead is pure waste. A WebSocket opens in a single HTTP upgrade handshake — typically sub-10 ms within a region — and streams PCM audio continuously thereafter.
Deepgram’s streaming API, ElevenLabs’ real-time TTS endpoint, and NVIDIA Riva’s gRPC streaming interface all accept raw PCM (L16, 16 kHz, mono) as input. This is the lingua franca of server-side audio processing, and WebSocket is the cheapest, simplest transport for it. The bandwidth cost is real: uncompressed 16-bit PCM at 16 kHz mono consumes 256 kbps, roughly 8–10× the bandwidth of Opus-encoded audio at equivalent quality. But between servers on 10 Gbps interconnects, this bandwidth overhead is irrelevant compared to the codec transcoding latency that a compressed format would introduce.
The WebSocket Audio Pipeline in Practice
A typical WebSocket-based voice AI backend in 2026 looks like this:
- The application server accepts an inbound call — either a WebRTC stream from a browser client bridged to PCM, or a SIP call from Twilio’s elastic SIP trunking, demodulated from G.711 μ-law to L16.
- The server opens a WebSocket to Deepgram for streaming ASR, sending 20 ms audio frames (320 bytes each at 16 kHz) as they arrive. Deepgram returns interim transcription hypotheses every 100 ms and a final hypothesis when end-of-speech is detected.
- The final transcription is sent to the LLM (OpenAI, Anthropic, or a self-hosted model via vLLM), which streams tokens back.
- As tokens arrive, the server opens a WebSocket to ElevenLabs for streaming TTS, sending text chunks and receiving PCM audio in return.
- The PCM audio is re-encoded to Opus or G.711 and forwarded to the client over WebRTC or SIP.
This architecture is the default in Vapi’s backend, in LiveKit’s agent pipeline when not using LiveKit’s own inference gateway, and in most custom voice AI deployments built on Pipecat, Daily’s open-source agent framework.
WebSocket’s Limitations
For all its simplicity, WebSocket has real limitations in the voice AI context. It runs over TCP, which means a single dropped packet stalls the entire stream until retransmission completes — head-of-line blocking that WebRTC’s UDP-based SRTP avoids entirely. It carries PCM audio, which at 256–512 kbps per stream becomes expensive at scale — a 1,000-concurrent-session deployment burns 256–512 Mbps in audio bandwidth alone before any application overhead.
WebTransport, the UDP-based QUIC successor standardized in the IETF, addresses both issues. It provides WebSocket-like APIs over an unreliable, multiplexed transport, eliminating head-of-line blocking while preserving browser-native support. The catch: Safari only added WebTransport support in early 2026, and Chrome’s implementation has been stable since 2023, but Firefox still lags. For backend-to-backend use, WebTransport is a stronger technical fit than WebSocket — Datagram over QUIC from a Go or Rust service to Deepgram’s edge would cut tail latency by 30–50 ms — but inertia and library maturity keep WebSocket dominant through 2026.
Real-Time Communication Stacks: WebRTC, SIP, PSTN
The voice AI transport ecosystem in 2026 spans three distinct protocol families, each occupying a different layer of the stack and optimized for a different segment of the network.
WebRTC: The Browser-to-Agent Default
WebRTC is the only browser-native protocol for real-time audio. It provides getUserMedia for microphone access, Opus encoding in the browser, echo cancellation, noise suppression, and automatic gain control — all enabled by default with no application code. When a user opens a voice AI interface in Chrome, Safari, or Firefox, WebRTC is the only viable path for capturing and transmitting audio.
In 2026, every major voice AI platform runs on WebRTC at the client edge. OpenAI’s ChatGPT Voice mode ship on WebRTC (originally on LiveKit’s infrastructure; now on an in-house WebRTC-LLM gateway). LiveKit Agents, Daily’s Pipecat framework, and Agora.io’s Conversational AI Engine all expose WebRTC as the primary client-facing transport. The pattern is consistent: a Selective Forwarding Unit (SFU) or media server sits between the browser client and the server-side AI agent, routing Opus frames with sub-50 ms latency.
WebRTC’s built-in codec support is a practical advantage. Opus, the mandatory-to-implement audio codec in WebRTC, delivers wideband speech quality (20 Hz–20 kHz) at 6–510 kbps, with adaptive bitrate negotiation that scales from 6 kbps in degraded network conditions to 128 kbps for stereo music. In production voice AI deployments, the typical client-side bitrate is ~20–32 kbps, with DTX (discontinuous transmission) disabled to avoid interfering with server-side VAD.
The WebRTC data channel — a separate SCTP-based channel alongside the media stream — serves a critical role in 2026 voice AI: it carries control events, function-call results, and visual context (base64-encoded still frames for multimodal agents). This is how an agent running on LiveKit can signal “the user interrupted” or “the function call returned a result” without embedding metadata in the audio stream itself.
SIP and the PSTN: Where Telephony Meets AI
Session Initiation Protocol (SIP) is the signaling protocol that powers the public telephone network. When a customer dials a phone number connected to a voice AI agent, their call traverses the PSTN as a circuit-switched audio stream, hits a carrier gateway, and arrives at the AI platform as a SIP INVITE with an attached media stream — almost always G.711 μ-law (PCMU) at 64 kbps.
The integration point is the SIP trunk. Twilio’s Elastic SIP Trunking, Vonage’s SIP trunking API, Plivo’s elastic trunks, and SignalWire’s programmable SIP all serve the same function: they bridge PSTN calls into IP-based media streams that a voice AI backend can consume. The SIP trunk provider handles carrier interconnects, number provisioning, STIR/SHAKEN attestation for caller ID, and the media transcoding between PSTN codecs (G.711, G.729) and whatever the AI backend expects (typically L16 PCM).
The latency penalty of the PSTN hop is non-trivial. A call from a mobile phone to a Twilio SIP trunk in the same metropolitan area adds ~50–100 ms of carrier routing. Cross-country PSTN routing adds 100–200 ms. International routing adds 200–400 ms. This is latency the voice AI pipeline cannot control — it sits upstream of the STT-LLM-TTS chain and is paid on every call, regardless of how fast the rest of the stack runs.
Codec Comparisons: Opus, PCMU, PCMA
The codec negotiation between the PSTN and the AI backend is a hidden source of audio quality degradation in many deployments.
| Codec | Bitrate | Bandwidth | Latency (encode) | Use Case |
|---|---|---|---|---|
| Opus | 6–510 kbps (typical 20–32 kbps voice) | 20 Hz–20 kHz (full-band) | 5–26.5 ms (frame sizes) | WebRTC client-to-server; LiveKit SFU routing |
| PCMU (G.711 μ-law) | 64 kbps | 300–3,400 Hz (narrowband) | <1 ms (no compression) | PSTN telephony; SIP trunks from Twilio, Vonage, Plivo |
| PCMA (G.711 A-law) | 64 kbps | 300–3,400 Hz (narrowband) | <1 ms (no compression) | European/Asian PSTN; SIP trunks outside North America |
| G.729 | 8 kbps | 300–3,400 Hz (narrowband) | 15 ms (algorithmic delay) | Bandwidth-constrained SIP trunks; SignalWire supports |
| L16 PCM (raw) | 256 kbps (16 kHz mono) | 0–8 kHz (wideband) | 0 ms | Backend-to-backend; Deepgram, ElevenLabs, NVIDIA Riva input |
The key insight: PSTN audio is narrowband. G.711, the universal PSTN codec, captures 300–3,400 Hz — roughly telephone quality. When a voice AI agent speaks through a phone call, the TTS engine may generate full-band Opus audio at 20 kHz, but any frequency content above 3.4 kHz is lost at the PSTN gateway. This means the audio quality of an AI agent over a phone call is inherently limited by the PSTN, not by the TTS model. Systems that serve both browser-based (WebRTC/Opus, full-band) and phone-based (SIP/G.711, narrowband) users should configure TTS output sampling rates accordingly: full-band for WebRTC, narrowband for SIP, to avoid wasting bandwidth on inaudible frequencies.
Twilio and Vonage both support Opus passthrough on their SIP trunks when the far-end carrier supports it, but in practice, G.711 remains the universal fallback and the de facto standard for PSTN voice AI in 2026.
Telephony Providers: The CPaaS Layer
The Communications Platform as a Service (CPaaS) layer sits between the PSTN and the voice AI backend. Choosing a provider means choosing a carrier network, a price per minute, a geographic footprint, and an API surface. The four providers that dominate voice AI evaluations in 2026 are Twilio, Vonage, Plivo, and SignalWire.
Twilio: The Safe Default
Twilio is the largest programmable communications platform in the world, and its dominance in developer-facing telephony is earned. The documentation is the best in the industry. The SDK coverage spans every major language and framework. The community support means that when an integration breaks, a Stack Overflow answer or GitHub issue thread almost certainly exists for the exact error.
For voice AI specifically, Twilio’s Elastic SIP Trunking supports G.711, G.729, and Opus codecs, with media gateway infrastructure distributed globally. Twilio’s number inventory is the deepest: local DIDs in 100+ countries, toll-free numbers, and short codes. Vapi, Retell AI, Bland AI, and virtually every voice AI platform ship with native Twilio integrations.
Twilio at a glance (US pricing, vendor-stated):
| Metric | Value |
|---|---|
| Inbound per-minute | ~$0.0085 |
| Outbound per-minute | ~$0.014 |
| Countries with DIDs | 100+ |
| Codecs | G.711, G.729, Opus, PCMU, PCMA |
| SIP trunking | Elastic SIP (origination + termination) |
| Voice AI integrations | Vapi, Retell AI, Bland AI, custom SIP |
The downside is price. Twilio is 30–50% more expensive per minute than Plivo for equivalent routes, and its pricing has more line items — phone number rental, recording storage, SIP trunk fees — which makes total cost of ownership harder to forecast than Plivo’s simpler pricing. At 100,000 minutes per month, the difference between Twilio and Plivo is ~$400–600 per month. At 1,000,000 minutes, it’s $4,000–6,000 — a meaningful operational cost.
Plivo: The Cost Leader
Plivo delivers Twilio-equivalent reliability at 40–60% lower per-minute pricing in Tier 1 markets (US, UK, Canada, Australia, Western Europe). Its core infrastructure in these markets is genuinely solid: call completion rates, audio quality, and latency are comparable to Twilio in most production deployments. Plivo’s API is clean, well-documented, and integrates with Vapi and other voice AI platforms that support bring-your-own-SIP-trunk configurations.
Plivo at a glance (US pricing, vendor-stated):
| Metric | Value |
|---|---|
| Inbound per-minute | ~$0.0042 |
| Outbound per-minute | ~$0.0085 |
| Countries with DIDs | 65+ |
| Codecs | G.711 (PCMU/PCMA), G.729 |
| SIP trunking | Elastic SIP trunking |
| Voice AI integrations | Vapi, custom SIP (fewer official guides than Twilio) |
Plivo’s limitations are geographic coverage and support tier. DID availability outside major markets is thinner, support response times are slower on lower-tier plans, and the community is smaller — unusual integration problems are less likely to have existing answers online. For high-volume, cost-sensitive deployments in well-covered markets, Plivo is a strong choice. For teams that need hand-holding or broad geographic reach, the savings warrant careful reconsideration.
Vonage: The Enterprise Choice
Vonage, operating as Vonage by Ericsson since the 2022 acquisition, occupies a different position. Where Twilio and Plivo are developer-first platforms, Vonage has a stronger enterprise sales and support motion: dedicated account managers, SLA-backed uptime guarantees, and a support experience comparable to traditional enterprise software vendors.
Vonage’s geographic coverage is its strongest differentiator. Its DID footprint extends into markets where Twilio and Plivo are thin — Eastern Europe, Southeast Asia, parts of Africa and the Middle East — making it the pragmatic choice for global deployments that need local numbers in less-served regions. Vonage’s API supports G.711, G.729, and Opus on its SIP trunks, and its media gateway infrastructure benefits from Ericsson’s carrier relationships and physical network presence.
The trade-off is developer experience. Vonage’s API documentation and SDK quality trail Twilio’s, and the platform’s enterprise orientation means pricing is negotiated rather than published — teams below ~$5,000/month in committed spend may find the onboarding friction disproportionate to the benefit.
SignalWire: The Programmable SIP Specialist
SignalWire occupies the niche for teams that need deep programmability without the CPaaS markup. Built by the original FreeSWITCH team, SignalWire offers a developer-centric SIP platform with a real-time API that exposes low-level call control: raw media access, custom codec negotiation, and programmable signaling that goes beyond what Twilio’s TwiML or Vonage’s NCCO abstractions allow.
For voice AI, SignalWire’s strength is its SIP interoperability. Teams that need to connect AI agents to existing on-premises PBX systems, legacy SIP phones, or custom PSTN interconnects find SignalWire’s lower-level APIs more flexible than the higher-level abstractions of Twilio or Vonage. Its per-minute pricing is competitive with Plivo’s, and its open-source lineage (FreeSWITCH under the hood) gives engineering teams full visibility into the media path.
SignalWire’s limitation is ecosystem breadth. It lacks the extensive partner integrations, pre-built voice AI platform connectors, and community knowledge base that Twilio and Vonage have accumulated. It is the right choice for teams that need programmable SIP primitives and have the engineering capacity to build the integration layer themselves.
Infrastructure-as-a-Service: WebRTC Platforms for Voice AI
Above the telephony layer sits the WebRTC infrastructure layer — the platforms that handle signaling, media routing, STUN/TURN, and agent orchestration. Three platforms dominate voice AI infrastructure evaluations in 2026.
LiveKit: The Voice AI Infrastructure Leader
LiveKit owns the voice AI infrastructure mindshare in 2026. OpenAI itself ran ChatGPT’s Advanced Voice mode on LiveKit’s infrastructure before building an in-house WebRTC gateway, and the platform’s open-source Apache 2.0 license means teams can self-host or use managed Cloud.
LiveKit’s architecture centers on a Selective Forwarding Unit (SFU): a media server that receives Opus frames from clients, routes them to the appropriate destinations (other clients, server-side agents), and never decodes or re-encodes the audio — preserving the low-latency properties of Opus end-to-end. The SFU model scales efficiently because the server’s CPU cost is proportional to the number of streams, not the number of participants multiplied by streams.
LiveKit Agents, the framework for building server-side voice AI on LiveKit, provides turn-detection plugins, STT/TTS/LLM service integrations (Deepgram, ElevenLabs, OpenAI, NVIDIA Riva), and function-calling support — all running as a long-lived process connected to the LiveKit room via WebRTC. The agent framework supports both the STT-LLM-TTS pipeline and speech-to-speech model paths, and it ships with a plugin architecture that lets teams swap components independently.
LiveKit Cloud pricing (vendor-stated, 2026):
| Tier | Monthly | Agent Session Minutes | Participant Minutes | Inference Credits |
|---|---|---|---|---|
| Build | $0 | 1,000 | Pay-as-you-go | $2.50 (~50 min) |
| Ship | $50 | 5,000 | 150,000 | $5 included |
| Scale | $500 | 50,000 | 1,500,000 | $50 included |
| Enterprise | Custom | Custom | Custom | Custom |
LiveKit’s per-minute cost for AI agent workloads lands around $0.01 per agent session minute plus $0.0004 per participant minute, with inference passed through at cost. Self-hosted LiveKit on Kubernetes crosses the profitability line at ~500 concurrent sessions — roughly 30–50% below managed Cloud pricing — but only for teams that already run Kubernetes in production and can absorb on-call operations.
Daily.co and Pipecat: The Open-Source Agent Framework
Daily.co is a WebRTC platform that has bet its voice AI strategy on Pipecat, the open-source agent framework it created and maintains. Pipecat supports both WebSocket and WebRTC transports, ships with native integrations for Deepgram (STT), ElevenLabs (TTS), and OpenAI (LLM), and runs as a Python or Node.js process that can be deployed anywhere — on Daily’s managed infrastructure, on AWS via Pipecat Cloud, or self-hosted.
Daily’s architectural differentiator from LiveKit is its transport abstraction. Pipecat agents can receive audio over WebRTC (from the Daily platform), WebSocket (from a custom client), or raw PCM (from a SIP-to-PCM bridge), and the agent code is the same regardless. This makes Pipecat the strongest choice for teams that need to serve browser-based, phone-based, and custom-client users from a single agent codebase.
Daily (and Pipecat when using Daily’s transport) uses Twilio and/or Cloudflare for its TURN infrastructure — a detail that matters because running a distributed TURN network is operationally expensive and offers little competitive differentiation. Daily’s decision to outsource TURN means its engineering focus goes into the agent framework, not into maintaining NAT-traversal infrastructure.
Pipecat Cloud, which reached general availability in January 2026, is a managed platform for deploying Pipecat agents on Daily’s global infrastructure with multi-region support and enterprise-grade SLAs. Amazon Bedrock AgentCore Runtime, announced in March 2026, supports Pipecat voice agents as a first-class deployment target — bringing Pipecat into the AWS ecosystem and giving enterprise teams a path to deploy voice AI on their existing AWS infrastructure.
Agora.io: Global Edge Presence for Pure RTC
Agora.io operates the Software-Defined Real-Time Network (SD-RTN), a proprietary UDP-based overlay network with edge nodes in 200+ regions. For pure real-time communication — peer-to-peer video calls, live streaming, interactive classrooms — Agora’s edge presence, particularly in APAC and LATAM, is unmatched by LiveKit or Daily.
Agora’s Conversational AI Engine is the company’s entry into the voice AI market. It bundles ARES ASR, an LLM gateway, and TTS into a single managed pipeline that runs on Agora’s SD-RTN, charging approximately $0.0265 per participant-minute (all-in, audio pipeline plus ASR, vendor-stated). Plain RTC audio minutes (no AI) cost $0.99 per 1,000 minutes — cheaper than LiveKit’s participant minutes at list.
The cost caveat with Agora is double-billing. If a voice AI session uses RTC audio minutes (for transport) and Conversational AI Engine minutes (for inference), Agora bills both. Standby or muted STT sessions incur charges. Recording costs are multiplied by participant count. The published per-minute rate is accurate, but the total invoice multiplies across service dimensions. Teams evaluating Agora for voice AI should model total cost as RTC audio minutes + AI Engine minutes + recording + signaling, not the AI Engine rate alone.
Agora’s pricing is also negotiated above 100,000 monthly minutes, which means teams cannot forecast 12-month spend from the pricing page alone. LiveKit’s published overage rates are fully transparent by comparison.
Latency Benchmarks: WebSocket vs WebRTC vs SIP
Latency in voice AI is measured across three dimensions: setup latency (time to establish a media session), transport latency (time for an audio packet to travel end-to-end), and pipeline latency (time from user stop-speaking to agent start-speaking). Each transport protocol performs differently across these dimensions.
Setup Latency
| Protocol | Typical Setup Time | What Happens |
|---|---|---|
| WebSocket | 5–20 ms (within region) | Single HTTP upgrade handshake; TLS already established |
| WebRTC | 200–800 ms | SDP offer/answer exchange, ICE candidate gathering, DTLS handshake, SRTP key exchange |
| SIP (PSTN inbound) | 500–2,000 ms | Carrier signaling, STIR/SHAKEN verification, media gateway negotiation |
WebSocket’s setup advantage is dramatic — a 10–50× improvement over WebRTC for the initial connection. This is why backend-to-backend connections use WebSocket: the overhead of a WebRTC handshake between servers in the same data center is unjustifiable. For browser clients, WebRTC’s setup latency is a one-time cost paid on session start, not on every audio frame, and is amortized over the duration of the call.
LiveKit’s WebRTC stack, tuned for voice AI, reduces ICE gathering time by pre-configuring TURN servers and signaling through a persistent WebSocket connection — cutting setup latency to ~200–400 ms in typical deployments. Agora’s SD-RTN, which uses proprietary UDP routing instead of standard ICE, claims sub-100 ms session establishment within its edge network (vendor-stated).
Transport Latency: Audio Frame Delivery
Once a session is established, transport latency measures the time from audio frame capture to delivery at the remote endpoint.
| Protocol | Typical One-Way Transport Latency | Jitter (variance) |
|---|---|---|
| WebRTC (Opus, same region) | 20–50 ms | ±5 ms |
| WebRTC (Opus, cross-continent) | 80–150 ms | ±20 ms |
| WebSocket (PCM, same region) | 10–30 ms | ±2 ms (TCP, no jitter buffer) |
| WebSocket (PCM, cross-continent) | 70–140 ms | ±10 ms (TCP retransmission adds variance) |
| SIP/PSTN (local call) | 50–100 ms | ±15 ms |
| SIP/PSTN (cross-country) | 100–200 ms | ±30 ms |
| SIP/PSTN (international) | 200–400 ms | ±50 ms |
WebSocket’s TCP-based transport delivers lower raw latency than WebRTC’s UDP-based SRTP on the same network path because it avoids the jitter buffer and packet reordering logic that WebRTC applies by default. However, this advantage comes at the cost of head-of-line blocking: one dropped TCP packet stalls the entire audio stream until retransmission, while WebRTC’s Opus decoder gracefully handles missing frames through packet loss concealment.
For browser-to-agent voice AI, the practical difference is minimal in good network conditions: both WebSocket and WebRTC deliver sub-50 ms transport latency within a region. In degraded conditions (packet loss above 2–3%), WebRTC’s resilience becomes noticeable — Opus with forward error correction and packet loss concealment maintains intelligibility at 10% packet loss, while PCM over WebSocket produces audible gaps at 1% packet loss.
Total Pipeline Latency: Production Numbers
The following are representative end-to-end latencies measured in production voice AI deployments in mid-2026, broken down by transport and pipeline architecture:
| Architecture | Setup | Transport | ASR | LLM | TTS | Total | Use Case |
|---|---|---|---|---|---|---|---|
| WebRTC + STT-LLM-TTS (LiveKit + Deepgram + GPT-4o + ElevenLabs) | 300 ms | 30 ms | 300 ms | 500 ms | 200 ms | 1,330 ms | Customer support voice agent |
| WebRTC + Speech-to-Speech (OpenAI Realtime on LiveKit) | 300 ms | 30 ms | — | 500 ms (combined) | — | 830 ms | Conversational AI, low-latency mode |
| WebSocket + STT-LLM-TTS (backend-to-backend, same region) | 10 ms | 20 ms | 150 ms | 400 ms | 150 ms | 730 ms | Server-side agent with browser bridge |
| SIP/PSTN + STT-LLM-TTS (Twilio trunk + Vapi) | 800 ms | 100 ms | 300 ms | 500 ms | 200 ms | 1,900 ms | Phone-based AI agent, local call |
| SIP/PSTN + STT-LLM-TTS (Vonage trunk + Vapi, international) | 1,000 ms | 250 ms | 300 ms | 500 ms | 200 ms | 2,250 ms | Phone-based AI agent, international |
The takeaway: browser-based voice AI over WebRTC cuts 600–900 ms of latency compared to phone-based voice AI over SIP/PSTN, almost entirely from eliminating the PSTN routing and carrier negotiation overhead. This is why OpenAI, Google, and Anthropic ship voice AI as native app features rather than phone numbers: the latency difference is perceptible and material to user experience.
Deployment Patterns: Edge, Cloud, and On-Device
Voice AI infrastructure in 2026 spans four distinct deployment patterns, each with different latency profiles, cost structures, and operational complexity.
Pattern 1: Fully Managed Cloud (Vapi, LiveKit Cloud, Agora AI Engine)
The simplest pattern: a managed platform handles the entire stack — telephony, WebRTC signaling, STT, LLM, TTS, and agent orchestration. The developer provides a prompt and a phone number; the platform handles everything else.
Vapi exemplifies this model. Developers configure a voice agent through Vapi’s dashboard or API, point a Twilio or Vonage SIP trunk at Vapi’s infrastructure, and Vapi handles the STT-LLM-TTS pipeline, turn detection, interruption handling, and function calling. Pricing is per-minute and all-inclusive. Vapi uses Deepgram for STT and ElevenLabs for TTS by default, with BYO-model options for both. The latency profile is the standard WebRTC + STT-LLM-TTS chain: ~1,000–1,500 ms end-to-end for browser calls, ~1,800–2,200 ms for PSTN calls.
LiveKit Cloud’s agent framework provides the same abstraction with more control: developers write agent logic in Python or Node.js using LiveKit’s agent SDK, which handles the transport, turn detection, and service integrations. The developer owns the LLM prompt and function definitions; LiveKit owns the infrastructure. This is the pattern OpenAI used before building in-house.
Best for: Teams that want to ship a voice AI product without building infrastructure. The managed platform premium is 2–4× the raw inference cost, but the engineering time saved is 10–20×.
Pattern 2: Self-Hosted WebRTC Gateway (LiveKit OSS, Pipecat + Daily Transport)
Teams that need infrastructure control without reinventing WebRTC run self-hosted LiveKit or self-hosted Pipecat with Daily’s transport layer.
Self-hosted LiveKit is the most common path for teams crossing 500 concurrent sessions. The LiveKit server (Apache 2.0, Go) runs on Kubernetes, with TURN servers (coturn or Cloudflare) providing NAT traversal. The agent processes run as sidecars or separate pods, connecting to the LiveKit SFU via WebRTC. Bandwidth egress costs drop to raw hosting-provider rates ($0.01–$0.09/GB on Hetzner or DigitalOcean vs $0.10–$0.12 on LiveKit Cloud), and the per-minute platform margin disappears. The trade is the engineering cost of running a distributed WebRTC infrastructure — Kubernetes expertise, TURN server maintenance, and on-call rotation.
Self-hosted Pipecat with Daily’s transport occupies a middle ground: the agent processes run on the team’s infrastructure, but the WebRTC signaling and media routing run on Daily’s managed platform. This decouples agent logic (which teams want to control) from WebRTC infrastructure (which teams want to outsource).
Best for: Teams running 500+ concurrent sessions with existing Kubernetes expertise, or teams that need to run AI inference on specific GPU hardware (e.g., NVIDIA A100s in a colocated data center for data residency compliance).
Pattern 3: Edge Deployment (NVIDIA Riva on Edge, Cloudflare Workers + WebRTC)
Edge deployment pushes STT and TTS inference to the network edge — GPUs in regional POPs or on-premises edge servers — to cut transport latency and keep audio data within a geographic boundary.
NVIDIA Riva, deployed as a NIM (NVIDIA Inference Microservice) on edge GPUs, achieves ASR latency of 80–120 ms and TTS first-audio latency of 50–100 ms (vendor-stated on A100). When deployed in the same metropolitan area as the user, total pipeline latency drops to 500–800 ms — competitive with speech-to-speech models running in centralized cloud regions. Riva’s gRPC streaming interface supports both WebSocket and raw gRPC transports, and its model catalog includes NVIDIA’s Nemotron streaming ASR model and FastPitch/HiFi-GAN TTS, both optimized for sub-100 ms latency on NVIDIA hardware.
The edge deployment pattern is most relevant for latency-sensitive use cases (real-time translation, live captioning, gaming voice chat) and for regulated industries (healthcare, finance) that require audio data to remain within a specific jurisdiction. The operational complexity is high: edge GPU infrastructure requires hardware procurement, colocation, or cloud-edge services (AWS Outposts, Azure Stack Edge, GCP Distributed Cloud Edge), and the model deployment pipeline must push updates to distributed endpoints.
Pattern 4: On-Device (Whisper.cpp, Mobile-Optimized TTS)
On-device inference bypasses the network entirely: STT and TTS run on the user’s phone, laptop, or embedded device, and only the text transcript travels over the network to the LLM.
Whisper.cpp, the C++ port of OpenAI’s Whisper model, runs real-time STT on Apple Silicon, Snapdragon 8 Gen 4, and Intel Meteor Lake processors with sub-50 ms per-chunk latency for the tiny and base models. Combined with on-device TTS (Apple’s AVSpeechSynthesizer, Google’s on-device TTS API, or StyleTTS 2 running via ONNX Runtime), the audio pipeline runs at 50–150 ms total — faster than any cloud-based STT+TTS chain.
The limitation is model quality. On-device Whisper tiny and base produce higher word error rates than cloud-based Deepgram Nova-2 or NVIDIA Riva, particularly in noisy environments and for accented speech. On-device TTS quality trails ElevenLabs and Play.ht for natural prosody and emotional expression. For use cases where latency is paramount and absolute accuracy is secondary (real-time dictation, in-car voice assistants, accessibility tools), on-device inference is the optimal pattern. For use cases where conversational naturalness matters (customer support, sales, therapy), cloud-based inference remains superior.
Failover Patterns
Production voice AI deployments in 2026 implement multi-layer failover to maintain uptime across components:
- Telephony failover: Twilio and Vonage both support automatic carrier failover — if the primary SIP trunk provider’s media gateway goes down, calls route through a backup provider. Plivo and SignalWire offer this as a configurable feature on enterprise plans. Implementation requires registering DIDs with multiple providers and configuring DNS SRV records for SIP failover.
- WebRTC infrastructure failover: LiveKit Cloud and Agora’s SD-RTN handle edge node failure transparently through automatic reconnection. Self-hosted LiveKit deployments implement failover through Kubernetes pod anti-affinity rules, multi-zone SFU deployment, and client-side reconnection logic that retries against a backup region.
- Model inference failover: Vapi, LiveKit Agents, and Pipecat all support model fallback chains — if the primary LLM (e.g., GPT-4o) returns an error or exceeds a latency threshold, the agent retries with a backup model (e.g., Claude 3.5 Sonnet, or a self-hosted model). STT and TTS fallback follows the same pattern: Deepgram → NVIDIA Riva → Whisper for ASR; ElevenLabs → Play.ht → on-device TTS for speech generation.
- Graceful degradation: The most sophisticated deployments implement tiered degradation. If the LLM response exceeds 800 ms, the agent plays a filler (“Let me think about that…”) while continuing generation. If TTS latency exceeds 500 ms, the agent drops to a faster, lower-quality TTS model for that turn. If the network degrades below 200 kbps, the WebRTC stack negotiates Opus down to 6 kbps to maintain connectivity.
AI voice infrastructure in 2026 has converged on a layered architecture: CPaaS providers (Twilio, Vonage, Plivo, SignalWire) handle the PSTN interface; WebRTC platforms (LiveKit, Daily.co, Agora.io) handle browser-to-agent transport; and model providers (Deepgram, ElevenLabs, NVIDIA Riva) handle the STT, LLM, and TTS inference. The integration points between these layers — SIP trunks, WebSocket bridges, WebRTC gateways — are where most production issues arise, and where latency accumulates.
The key architectural decisions for teams building voice AI in 2026 are:
- Transport: WebRTC for browser clients (Opus at ~20–32 kbps, sub-50 ms transport latency). WebSocket for backend-to-backend (PCM at ~256 kbps, sub-20 ms setup). SIP/PSTN only when phone numbers are a hard requirement — and accept the 600–900 ms latency penalty.
- Pipeline: STT-LLM-TTS for flexibility and model composability. Speech-to-speech for lowest latency at the cost of vendor lock-in. The gap between the two architectures is narrowing, and 2027 will likely see production-grade open-weight speech-to-speech models that combine the latency advantages of end-to-end models with the composability of pipeline architectures.
- Infrastructure: Fully managed (Vapi, LiveKit Cloud) for speed to market. Self-hosted (LiveKit OSS, Pipecat) for cost control above 500 concurrent sessions. Edge deployment (NVIDIA Riva) for latency-critical and data-residency use cases. On-device for privacy-maximal, latency-minimal applications where model quality can be compromised.
- Provider: Plivo for cost-sensitive Tier 1 deployments. Twilio for ecosystem breadth and documentation quality. Vonage for global geographic coverage and enterprise SLAs. SignalWire for deep SIP programmability. LiveKit for voice AI infrastructure (the default in 2026). Daily.co/Pipecat for transport-agnostic agent frameworks. Agora for APAC/LATAM edge presence in pure RTC workloads.
The voice AI infrastructure market is maturing rapidly. In 2024, every team built their own WebSocket proxy to bridge Twilio SIP trunks to Deepgram and ElevenLabs. In 2025, platforms like Vapi and LiveKit Agents abstracted that integration into a single API call. In 2026, the abstraction extends to deployment: Pipecat Cloud and Amazon Bedrock AgentCore Runtime make deploying a production-grade voice agent as simple as deploying a web application. The frontier in 2027 will be WebRTC-LLM gateways — purpose-built infrastructure that eliminates the SFU model’s unnecessary signaling renegotiations and network hops, replacing them with a direct UDP path from the browser to the inference backend, tuned specifically for the latency and loss characteristics of AI-generated speech.
This article was researched and written in July 2026. Pricing, latency numbers, and feature availability reflect vendor-stated data and independent production measurements as of the publication date. Verify current numbers with each vendor before making infrastructure commitments.
For a deeper look, see our guide on Voice AI Agents vs. Traditional IVR:.
For a deeper look, see our guide on AI Call Bots: The Business Guide.
For a deeper look, see our guide on Understanding LATA, IntraLATA, and InterLATA Calls.
For a deeper look, see our guide on 50+ VoIP Features and Terms Explained.
You can also run the numbers with our AI receptionist ROI calculator.
For a deeper look, see our guide on how to read a business phone bill.
For a deeper look, see our guide on how to set up an AI phone receptionist.
For a deeper look, see our guide on AI Voice Agent Pros & Cons:.
For a deeper look, see our guide on Best AI Communication Software for Business.
2026 market context for AI voice infrastructure
In late 2026 buying cycles, AI voice projects fail less on model quality and more on whether the telephony path, recording chain, and identity plane were designed as production systems.
Signals that reshape architecture priorities
- FCC (June 30, 2025 data; May 2026 release): Business interconnected VoIP ~44.0M (+4.1% YoY); ~83.6% of business fixed voice; total fixed VoIP ~63.4M vs ~15.0M switched—AI voice attaches to IP dial plans (FCC Voice Telephone Services).
- Wireline share shrinks: FCC three-year view shows wireline technologies under ~17% of ~473M retail voice connections; mobile remains the volume majority—design for softphone and mobile handoff, not lobby hardware assumptions.
- Metrigy UCaaS: $23.0B in 2025 (+6.1%); ~58.6% of businesses use UCaaS as their sole platform; seats ~117.1M (+6.9%); forecast ~$28.0B by 2030 (~4% CAGR) (Metrigy).
- Hybrid perimeter: Gallup ~52% hybrid among remote-capable workers—WebSocket/SIP clients on laptops and phones are first-class endpoints (Gallup).
Readability checklist for production teams
- Write latency budgets (media, STT, LLM, TTS) before you pick a vendor logo.
- Separate CPaaS trunks from UCaaS seats in diagrams so cost and ownership stay clear.
- Require recording/compliance paths in the same design review as the speech model.
- Load-test concurrent sessions on the real WAN—not a lab Wi-Fi demo.
Teams that treat AI voice as infrastructure ship calm cutovers. Teams that treat it as a chat widget usually rediscover SIP, e911, and observability after the first outage ticket.
What the latest data shows
AI voice infrastructure buying in 2026 is constrained by latency budgets and telephony integration quality—not model demos alone.
Verified signals
- Gartner-cited contact-center forecasts still treat full automation as roughly 1 in 10 interactions in 2026; most value is assistive AI on live agent paths (industry citations).
- Business fixed voice rides interconnected VoIP at scale: 44.0 million U.S. business subscriptions as of June 2025 (+4.1% YoY) (FCC).
- Streaming STT/TTS over WebSockets only works if SIP/PSTN handoffs, recording, and e911 stay first-class—not bolted on later.
What to do with this
- Benchmark end-to-end latency (STT → intent → TTS) on a pilot trunk before production.
- Require recording retention and warm-transfer APIs in the same RFP as model accuracy.
