Skip to content
Docs menu

Realtime voice agents (design)

The design for phone and browser voice agents on the harness, such as a receptionist or a collections agent, and where each part will run. Not built yet.

View .md

Venom's voice agent framework will run AI phone agents on @venom/harness. A call goes through a cascaded, full-duplex pipeline:

caller ⇄ transport ─► VAD / turn detector ─► streaming STT ─► harness agent loop (business tools)
                ▲                                                     │ text deltas
                └──────────── pacer ◄── streaming TTS ◄── speech guard ◄── sentence chunker

Scope

In scope for the first round:

  • the framework, @venom/voice: pipeline, turn-taking, transports, compliance and a call store; the orchestrator's realtime transcription endpoint; calls as a session type in venom serve;
  • three transports: self-hosted SIP through Asterisk, Twilio Media Streams, and a browser test line in the console;
  • two example agents: a receptionist (booking and FAQ tools) and a collections agent (account lookup and payment-plan tools, with a collections rules pack);
  • a Calls page in the console with live and past calls, transcripts, per-turn latency and compliance events.

Not in scope: speech-to-speech models as the reasoning step, a campaign dialer (outbound calls go one at a time through the API), realtime relayed through Venom Cloud, video and SMS.

Where things run

Piece Process Why
Call runtime: transports, voice activity detection, turn detection, chunker, speech guard, pacer, compliance venom serve, through @venom/voice A call is a long-lived session with an agent loop, which venom serve already hosts.
Agent loop and business tools the same process, one harness session per call The permission gate, audit, guards and idempotency keys apply to business tools unchanged.
Streaming speech-to-text the orchestrator: GET /v1/realtime?intent=transcription over a WebSocket Keys stay in the orchestrator, aliases and fallback work as for other endpoints, usage is priced and logged.
Text-to-speech the orchestrator: POST /v1/audio/speech The existing voice contract.
Speech models local servers by default, or a cloud provider behind an adapter Local first.
SIP, trunks, codecs, transfer Asterisk, self-hosted venom does not implement a SIP user agent.

Timing budget

The target is under 800 ms from the end of the caller's speech to the first agent audio. Each turn will record its timings relative to the end of speech: the endpointing decision, the final transcript, the first text token, the first speakable segment and the first audio byte.

For app builders

Because a call is a harness session, a voice agent is an ordinary venom agent with business tools and a speech front end. An app such as a collections service can reuse its existing tools, permission rules and audit trail.