---
title: Realtime voice agents (design)
description: The design for phone and browser voice agents on the harness, such as a receptionist or a collections agent, and where each part will run. Not built yet.
section: reference
order: 220
audience: [builder, operator]
sources: [docs/voice-agents.md]
features: [reference.doc-voice-agents]
---

> [!NOTE]
> This is a design (ADR 0043). The realtime agent framework is not built yet. The speech endpoints it builds on, `POST /v1/audio/speech` and `GET /v1/audio/voices`, ship today and are covered in the [Voice section](/docs/voice.md).

Venom's voice agent framework will run AI phone agents on `@venom/harness`. A call goes through a cascaded, full-duplex pipeline:

```text
caller ⇄ transport ─► VAD / turn detector ─► streaming STT ─► harness agent loop (business tools)
                ▲                                                     │ text deltas
                └──────────── pacer ◄── streaming TTS ◄── speech guard ◄── sentence chunker
```

## Scope {#scope}

In scope for the first round:

- the framework, `@venom/voice`: pipeline, turn-taking, transports, compliance and a call store; the orchestrator's realtime transcription endpoint; calls as a session type in `venom serve`;
- three transports: self-hosted SIP through Asterisk, Twilio Media Streams, and a browser test line in the console;
- two example agents: a receptionist (booking and FAQ tools) and a collections agent (account lookup and payment-plan tools, with a collections rules pack);
- a Calls page in the console with live and past calls, transcripts, per-turn latency and compliance events.

Not in scope: speech-to-speech models as the reasoning step, a campaign dialer (outbound calls go one at a time through the API), realtime relayed through Venom Cloud, video and SMS.

## Where things run {#placement}

| Piece | Process | Why |
|---|---|---|
| Call runtime: transports, voice activity detection, turn detection, chunker, speech guard, pacer, compliance | `venom serve`, through `@venom/voice` | A call is a long-lived session with an agent loop, which `venom serve` already hosts. |
| Agent loop and business tools | the same process, one harness session per call | The permission gate, audit, guards and idempotency keys apply to business tools unchanged. |
| Streaming speech-to-text | the orchestrator: `GET /v1/realtime?intent=transcription` over a WebSocket | Keys stay in the orchestrator, aliases and fallback work as for other endpoints, usage is priced and logged. |
| Text-to-speech | the orchestrator: `POST /v1/audio/speech` | The existing voice contract. |
| Speech models | local servers by default, or a cloud provider behind an adapter | Local first. |
| SIP, trunks, codecs, transfer | Asterisk, self-hosted | venom does not implement a SIP user agent. |

## Timing budget {#timing}

The target is under 800 ms from the end of the caller's speech to the first agent audio. Each turn will record its timings relative to the end of speech: the endpointing decision, the final transcript, the first text token, the first speakable segment and the first audio byte.

## For app builders {#builders}

Because a call is a harness session, a voice agent is an ordinary venom agent with business tools and a speech front end. An app such as a collections service can reuse its existing tools, permission rules and audit trail.

## Related

- [Voice](/docs/voice.md)
- [Design decisions (ADRs)](/docs/reference/adr.md)
