Realtime transcription
Stream audio to the orchestrator over a WebSocket and get transcripts back as you speak, with OpenAI's realtime transcription events, in front of local or cloud speech models.
GET /v1/realtime?intent=transcription is a WebSocket. You send audio as you capture it and get the
transcript back while the caller is still talking. It speaks OpenAI's realtime transcription events, so an
OpenAI client works against it unchanged, and the orchestrator routes it like any other request: aliases,
fallback, breakers, key budgets, the request log and per-second pricing. The voice agents' streaming
speech recognition (ADR 0043) uses it.
Connect
Open ws://127.0.0.1:8787/v1/realtime?intent=transcription&model=venom-stt-stream with your client key in
Authorization: Bearer <key>. A browser-style client can send the key as the subprotocol
openai-insecure-api-key.<key> instead. Keys never go in the URL: a key or api_key query parameter is
refused. Leave model out and the orchestrator routes on the model of your first session.update.
The server sends session.created at once. Your first message must be session.update, within 5 seconds:
{"type": "session.update",
"session": {"type": "transcription",
"audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000},
"transcription": {"model": "venom-stt-stream", "language": "en"},
"turn_detection": null}}},
"venom": {"partials": "replace"}}Formats: audio/pcm (16-bit little-endian mono at 8000, 16000 or 24000 Hz), audio/pcmu and audio/pcma
(G.711, 8 kHz, as phone lines carry it). The orchestrator converts to whatever the backend takes. Once the
backend's session is up you get session.updated, with venom.upstream and venom.model naming the target.
Stream
| You send | What it does |
|---|---|
input_audio_buffer.append |
{"audio": "<base64>"}, 20 to 100 ms per message, at most 1 MiB |
input_audio_buffer.commit |
the end of the caller's turn: the backend finalizes the item |
input_audio_buffer.clear |
drops the audio not yet committed |
| You get | Meaning |
|---|---|
input_audio_buffer.committed |
an item ended (your commit, or the backend's own VAD) |
conversation.item.input_audio_transcription.delta |
new text for the item; deltas only ever append |
venom.transcription.partial |
with venom.partials: "replace": the whole current guess, and stable_len, the part that won't change |
conversation.item.input_audio_transcription.completed |
the item's final transcript |
input_audio_buffer.speech_started / speech_stopped |
from backends that run VAD, with server_vad |
error |
{type, code, message, retryable} |
Some backends revise their partial results (FunASR's 2pass mode first guesses, then corrects each sentence). An OpenAI-only client gets deltas for the corrected text only; ask for venom partials to see the guesses too.
Failures
The orchestrator falls back to the next target only while the session starts. After session.updated, an
upstream failure sends an error with retryable: true and closes with code 1011, and your client
reconnects. 1013 means no upstream could take the session; 4408 means no session.update arrived in time;
4400 means a bad first message or model.
Audio is never held back: each frame goes to the backend as it arrives. If the backend stops reading, the orchestrator drops audio beyond one second rather than queueing it, so a slow backend never stalls your client.
Backends
| Kind | Backend |
|---|---|
funasr |
FunASR's funasr-wss-server-2pass, local, no key (base_url: ws://host:10095) |
openai |
OpenAI's realtime transcription: gpt-live-transcribe, gpt-transcribe, gpt-realtime-whisper (and the older gpt-4o-transcribe, whisper-1) |
whisperlive |
a WhisperLive server, local, no key (base_url: ws://host:9090) |
elevenlabs |
ElevenLabs scribe_v2_realtime; phone audio passes through as µ-law |
dashscope |
Alibaba Model Studio: qwen3-asr-flash-realtime (Singapore and Beijing), fun-asr-realtime, paraformer-realtime-v2 / -8k-v2 (Beijing keys only) |
For a local FunASR server, docker compose --profile asr up -d funasr starts one on 127.0.0.1:10095 (it
downloads its models from ModelScope the first time). docs/configuration.md has the upstream example. Each session is one row in the request log, with the audio
in audio_ms and its cost at the model's per_second. Venom Cloud doesn't relay WebSockets yet: point the
client at a local orchestrator or a daemon it can reach.