Production voice AI needs scheduling, not bigger models

When OpenAI published its engineering account of running voice AI at scale, the striking thing was the subject matter: Kubernetes deployment, UDP port exhaustion, connection handshakes when calls move between machines. And no latency figures. The largest model company in the world had published a distributed-systems paper about voice latency. 

The same shift appears elsewhere. ElevenLabs foregrounds its orchestration engine. LiveKit focuses on session lifecycles, workers, and interruption handling. Even model providers now make latency configurable: OpenAI’s real-time models offer five reasoning-effort levels, from minimal when speed matters most to xhigh when additional reasoning justifies the delay. 

The tempting conclusion is that infrastructure is replacing models. It isn’t. A benchmark measuring whether voice agents complete tasks improved from 30% to 67% in eight months through model changes alone. Models still drive outcomes. But production performance is now shaped by a second axis: how inference, state, and real-time interactions are coordinated. That is where engineers spend their days. 

Does orchestration matter more than inference? 

A Voice AI system is usually drawn as a chain: detect that the user has stopped speaking, transcribe, generate a reply, synthesise speech. Production systems do not wait for each stage to finish. Recognition streams while the user speaks. Context retrieval and tool calls can begin before the reply exists. The language model can start generating before the turn is confirmed over. That’s a speculative generation. 

This overlap makes orchestration a scheduling problem: what starts early, what runs in parallel, what waits, and what gets cancelled. Only a few dependencies are fixed:  

• audio must play in order;  

• a sentence must exist before synthesis; 

• a tool must return before its result can be reported.  

Endpointing, speculative generation, streaming, interruption policy, and tool timeouts are design choices. 

So does orchestration matter more than inference? Not universally. In cascaded agents, the language model can account for 300–1000 ms of latency, while endpointing adds 200–800 ms. The two are coupled: speculative generation trades latency for wasted tokens, while faster recognition can reduce endpointing delay. The distinction is simpler:  

• inference determines what the agent can do;  

• orchestration determines how that work happens in real time.  

As Voice AI moves into production, that scheduling problem becomes a performance constraint in its own right. 

Scheduling becomes the product 

Users judge voice agents by how naturally they respond, handle interruptions, and keep conversations moving. These outcomes depend on how the system schedules competing tasks, creating five problems that dominate production Voice AI: 

1. Asynchronous execution. Pipecat runs stages concurrently, with a separate fast lane for control events like ‘stop talking’. If a downstream stage is slower, its queue backs up. A buffer before synthesis cannot hide the gap because the audio does not exist yet. Underruns therefore cause silence, so synthesis must start earlier or use smaller chunks. LiveKit documents no equivalent model, making behaviour under load difficult to compare. 
2. Speculative generation. The LLM starts generating from a partial transcript before the turn is confirmed. If the turn changes, the response is discarded and regenerated, increasing token use. A mispredicted CPU branch never retires. A speculative voice response can reach the user before the underlying tool call completes. Without a rollback barrier, speculation becomes talking early. The cost is also different: wrong branches waste compute, and wrong turns waste tokens. One report recorded 14,858 tokens across two requests for a single turn, although most of the second request was cache-served. 
3. Cancellation. When a user interrupts an agent, work already in flight has to be stopped or safely discarded across multiple systems. Pipecat and LiveKit Agents both handle cancellation within the pipeline, but their tool-call semantics differ. At the tool boundary, both provide concurrency controls such as timeouts and cancellation flags rather than transaction management. LiveKit also provides duplicate suppression, but it is off by default, matches on tool name rather than arguments, and applies only while the first call is still running. It therefore does not provide cross-turn idempotency. The frameworks make opposite default choices on interruption: Pipecat cancels an active tool call, while LiveKit lets it finish and discards the result. The engineering risk is that changing frameworks silently changes what happens to work already in flight. 
4. Interruptions. Barge-in is where cancellation and speculative generation collide, and it exposes the most consequential scheduling failure: the system can lose state while work is still executing. In October 2025, a bug report against a widely used voice-agent framework documented a caller interrupting an agent just after it placed an order. The system placed the same order twice. The first tool call had succeeded, but its result never reached the conversation history. On the next turn, the model had no record of the completed action and issued it again. The bug was fixed in July 2026 by committing tool messages to history, but the failure mode matters more than the fix. 
5. Concurrency. One call is a scheduling problem; a thousand concurrent calls are a capacity-management problem. The frameworks expose incompatible failure policies: Vapi queues callers, Retell holds them and then diverts, while Pipecat Cloud rejects immediately with HTTP 429.  

What comes next 

Production Voice AI is increasingly a scheduling problem: transport, session state, turn-taking, cancellation, tools, admission control, and draining sit outside the model. Yet the boundary is moving, with turn detection, interruption handling, and tool execution increasingly absorbed into models. 

What remains unproven is whether better scheduling produces better performance at scale. Published benchmarks rarely measure the full stack under concurrent load. The missing metric is straightforward: P99 conversational latency against concurrent-session count. Until that number exists, we can say where the engineering complexity sits but not where the performance gains come from. 

Disclaimer: This is a personal blog. The views and opinions expressed in this article are those of the author and do not represent those of people, institutions, or organizations that the author may or may not be associated with in a professional or personal capacity. All information is provided on an as-is basis. 

About the Author 

Mikita Hanusenka is a senior engineering leader with 10+ years of experience who built and scaled a cross-platform real-time communication platform for the gaming industry, enabling voice and video interactions for millions of concurrent users with ultra-low-latency performance.