Published · ICANN 2026 · Springer LNCS
Latency Optimization in Production Voice AI Pipelines: A Systems Approach
Voice assistants feel slow when every step waits for the one before it. We measured a production voice pipeline end to end and found the cheapest fix was not a faster model. It was letting two steps run at the same time.
- median end-to-end latency, 3,277 ms to 1,909 ms
- −41.8%
- milliseconds to the first audio the caller hears
- 1,293
- lines of middleware changed for the main gain
- ~10
- runs per configuration, on real production API calls
- 30
Why voice feels slow
A spoken reply passes through several services in a row. The words are recognised, a language model works out what the person wants, and a speech engine turns the answer back into sound. Each hop adds its own wait: a network round trip, the model's time to its first token, the speech engine's time to its first byte. Past about two seconds, a conversation starts to feel broken.
What we measured
The pipeline ran ElevenLabs for speech synthesis, Anthropic's Claude for intent detection and a NestJS middleware over WebSockets. We put timestamps at six boundaries across five stages and ran 30 benchmark sessions per configuration against the real production APIs, then tested five changes: streaming the speech in chunks, running intent detection and synthesis concurrently, compressing the prompt, caching session state, and segmenting the audio adaptively.
What we found
No single service was the problem. The biggest cost was waiting: synthesis started only after intent detection had finished. Starting both together took about ten lines of middleware and cut the median end-to-end time from 3,277 ms to 1,909 ms, a 41.8 percent reduction. The mean fell by 1,379 ms (95% confidence interval 1,185 to 1,573 ms), and the spread narrowed as well.
Streaming the first chunk of speech brought the time to first audio down to 1,293 ms, so the caller hears the start of the answer sooner. Cutting the prompt from about 20 KB to about 3 KB removed roughly 85 percent of the input tokens and about the same share of the cost per call, with no measured loss in accuracy, though it saved little time on its own.
What it means for your system
Before paying for a faster model or another provider, measure every boundary and look for steps that wait on each other. Removing a serial wait tends to gain more than speeding up any single stage, and here it took a change of about ten lines.
What the study does not show
- It covers one production pipeline, with 30 runs per configuration.
- Speech-recognition time sits inside the synthesis provider's first-byte time and is not measured separately.
- Three of the five changes are estimated from a latency model rather than measured one by one; the 41.8 percent comes from the concurrency change alone, measured on its own.
A voice or AI pipeline that feels slow?
We measure it first, then fix the waits that cost the most. The assessment ends with a written plan you keep.
Start a project