Latency is an ownership problem
Voice-agent latency is a chain of accountable decisions—not a flattering average on a dashboard.
- problem
- Voice-agent latency crosses asynchronous stages, so aggregate timing hides who owns delay, pressure, cancellation, and failed recovery.
- scope
- A public operating model for stage budgets, queue pressure, interruption, degraded modes, and per-turn traces without customer-specific details.
- environment
- LiveKit-style real-time voice runtimes with streamed STT, LLM, tools, TTS, and transport boundaries.
Assumptions
- Every stage can emit correlated telemetry under one turn identity.
- Cancellation and degraded-mode state can cross provider boundaries.
Limitations
- Customer topology, credentials, recordings, and production metrics are intentionally withheld.
- This is an ownership model, not a provider benchmark or a universal latency target.
Table of contents 4 sections
Define the path the user experiences
A latency number without ownership is theater. The measured path follows what the user actually experiences: audio ingress, VAD endpointing, STT, prompt and memory assembly, LLM first token, tool interception, TTS first audio, transport readiness, and playout.
- Audio ingress and endpointing
- Context, model, and tool execution
- Speech synthesis, transport, and playout
- Interruption, cancellation, and recovery
- ingress audio frames, VAD evidence, endpoint decision
- understand STT, prompt assembly, memory, model first token
- act tool boundary with timeout, fence, and outcome state
- speak TTS first audio, transport readiness, playout
- recover cancellation, fallback, degraded mode, final trace
Give every stage an owner and pressure policy
Every stage needs a stable owner, a bounded queue or explicit pressure policy, and a telemetry name that survives internal refactors. Otherwise the dashboard can look fast while endpointing drifts, providers stall, queues age, or the agent keeps speaking after the user interrupts.
type StageContract = { owner: string; timeoutMs: number; onTimeout: 'cancel' | 'degrade' | 'fallback'; traceName: string;};Correlate the operational view per turn
The operational view stays per-turn and correlated: stage percentiles, queue pressure, provider fallback, interruption detection, audio cutoff, and the degraded path when voice output is unhealthy. Averages are where bad conversations go to hide.
| boundary | evidence | failure question |
|---|---|---|
| endpointing | p50 / p95 / p99 + decision reason | Did silence or noise end the turn? |
| model | first token + cancellation state | Did stale work survive interruption? |
| speech | first audio + playout start | Did generated audio reach the user? |
| recovery | fallback + degraded-mode trace | Did the system fail visibly and safely? |
Publish the method without inventing proof
This public note deliberately omits customer context, credentials, private topology, and unsupported production metrics. The method is the proof: assign the budget, instrument every boundary, preserve cancellation, and make recovery a first-class path.
Replace aggregate latency with owned stage traces
- observation
- A single end-to-end number can improve while endpointing, queue age, or interruption behavior gets worse.
- hypothesis
- Stable stage ownership and one correlated turn identity will make hidden pressure and recovery failures attributable.
- change
- Name each boundary, emit stage spans and queue state, and carry cancellation and fallback decisions through the same turn trace.
- measurement
- The public verification target is coverage: every stage, pressure transition, interruption, fallback, and degraded path must appear in one trace. No private latency numbers are claimed here.
- mechanism
- Correlation preserves causality across asynchronous providers, while explicit queue and cancellation state prevents a healthy average from hiding a broken conversation.
- decision
- Keep the owned-stage model; reject uncorrelated averages as the primary operational view.
- field-observed Production failure vocabulary
Endpoint drift, provider stalls, queue age, interruption cutoff, fallback, and degraded voice output are treated as distinct operational states.
- implemented Per-turn trace contract
Stage owners, queue pressure, cancellation, fallback, and recovery share one correlated telemetry path.
- reference LiveKit Agents documentationopen ↗
Public platform reference for the real-time agent runtime discussed by this field doctrine.