A real-time voice AI agent connects telephony or browser audio to speech recognition, conversation orchestration, approved tools and speech playback. The difficult work is coordinating turns, interruptions, latency and business actions. A fluent spoken answer does not establish correct identity, evidence or permission to act.

This reference architecture describes a modular ASR–LLM–TTS path and the operating controls around it. Integrated speech-to-speech services can combine stages, but the application still needs explicit session state, authorization, handoff and recovery behavior.

Voice AI agent architecture

Caller / browser microphone
    → telephony or media gateway
    → authenticated audio session
    → streaming ASR + turn detection
    → conversation state + permitted evidence
    → LLM proposes response or bounded tool request
    → independent tool authorization and validation
    → text-to-speech + controlled playback
    → caller

Across the path: interruption control, session events,
human handoff, minimal telemetry and failure recovery

ASR means automatic speech recognition; TTS means text-to-speech. In a modular path, each stage exposes useful control and diagnostics but adds integration work. An integrated realtime service can reduce application plumbing while changing which stage-level controls and events are available. Evaluate its documented contract rather than assuming identical behavior.

Azure Voice Live guidance documents a realtime audio service and configuration choices. The architecture here is provider-neutral and does not claim an SDK integration was executed.

Keep media transport and call control distinct

A media stream carries audio; call control determines connection, transfer and termination. Validate the provider’s supported direction, encoding, sample rate and event semantics. A receive-only stream cannot necessarily play a synthesized answer back to the caller.

Twilio’s Media Streams message reference distinguishes stream events and supported messages for sending media, marking playback and clearing buffered audio. Implement the exact selected interface rather than assuming any WebSocket can control the call.

Bind media sessions to trusted call metadata and application scope. Authenticate callbacks or connections using the provider’s documented mechanism. Do not treat a caller-supplied account number or a speaker label as proof of identity. Use an approved verification flow before exposing account-specific information.

Turn detection and interruption handling are state management

Voice activity detection indicates likely speech. End-of-turn detection decides when to respond. Barge-in handling decides what to do when the caller speaks during agent playback. Background noise and short acknowledgments can resemble interruption attempts, so evaluate them separately.

LiveKit’s interruption documentation describes distinguishing interruption attempts from non-interruptive speech. This illustrates why acoustic activity and conversational intent are not identical signals; behavior varies by implementation.

Cancel all relevant output, not only the next token

When an interruption is accepted, coordinate generation cancellation, queued TTS and buffered playback according to the transport contract. Otherwise stale audio can continue after the application believes it stopped. Record what was actually played, not merely what the model generated.

Preserve conversation consistency. If the caller interrupts before hearing a proposed appointment time, the next turn should not assume it was communicated and accepted. Keep spoken, generated and approved action state distinct.

Measure a user-facing latency budget

The useful measure is the delay from the caller’s relevant speech to an appropriate audible response. Capture and buffering, turn detection, recognition, retrieval, model processing, tool calls, synthesis and playback all contribute.

Record first partial text, finalized turn, first generated response, first audible output and completion where observable. Compare latency distributions under relevant concurrency. A low model inference time does not prove that the conversation feels responsive.

Stream output where supported, but avoid speaking unvalidated or sensitive content merely to save milliseconds. Some actions require confirmation before execution. Set task-specific objectives and test trade-offs instead of inventing a universal response-time promise.

Keep tool authorization outside the language model

Expose narrow tools with explicit arguments and constrained outcomes. The model can propose looking up an appointment; the backend decides whether the caller may view or change it. Validate record ownership, tenant scope and business rules at execution time.

Use stable request identifiers and idempotency for retried side effects. Bind any required approval to the concrete action and target. A spoken “yes” must be interpreted within the active, verified proposal rather than reused as generic permission for later operations.

Apply the guardrails design to output and actions. Treat retrieved documents and tool responses as data, not instructions that may widen privileges.

Design human handoff before launch

Define triggers such as an explicit caller request, repeated recognition failure, unavailable evidence or a task outside the approved scope. The route must identify an available destination and a fallback when transfer fails.

Pass only the permitted context needed for continuity: verified identity status, task summary, completed actions and unresolved questions. Mark uncertain transcript content clearly. Avoid making the caller repeat everything, but do not transmit an unrestricted conversation dump by default.

Make transfer state observable. An attempted transfer is not the same as a connected human conversation. Test busy destinations, disconnects and after-hours behavior with the responsible operations team.

Protect audio and every derivative

Review recording notice, permitted use, access and retention with the appropriate owners for the operating locations. Audio, transcripts, summaries and embeddings can all contain sensitive information. Do not infer compliance from a vendor’s marketing label.

Minimize unrestricted trace payloads and protect stored recordings and playback links. Define deletion and retention behavior across downstream stores. Review telephony, speech, model and tool providers as a complete data flow.

Release checklist for a voice agent

1. Test relevant languages, accents, noise and channel conditions.
2. Measure partial and final recognition, speaker attribution where used, and task outcomes.
3. Exercise pauses, interruptions, false interruptions and queued playback cancellation.
4. Test denied record access and unapproved tool actions.
5. Replay requests to verify side-effect handling.
6. Simulate tool timeouts, network loss and provider errors.
7. Confirm human transfer and failure fallback.
8. Measure latency, cost and review effort under representative load.
9. Version configuration and keep a rollback and operating owner.

For speech-specific design, read real-time versus batch transcription. Use the testing playbook for generated responses and action boundaries.

Common questions

Build conversation reliability around explicit boundaries

For related AI application choices, read RAG versus fine-tuning and prompt engineering. To review voice orchestration, tool controls and recovery, discuss your production AI reliability roadmap.