---
title: Realtime transcription
description: Stream audio to the orchestrator over a WebSocket and get transcripts back as you speak, with OpenAI's realtime transcription events, in front of local or cloud speech models.
section: voice
order: 25
audience: [operator, builder, agent]
sources: [docs/architecture.md, docs/adr/0043-realtime-voice-agents.md, docs/voice-agents.md, services/go-orchestrator/internal/realtime, services/go-orchestrator/internal/voice/realtime.go, services/go-orchestrator/internal/audio]
features: [orchestrator.api-realtime]
---

`GET /v1/realtime?intent=transcription` is a WebSocket. You send audio as you capture it and get the
transcript back while the caller is still talking. It speaks OpenAI's realtime transcription events, so an
OpenAI client works against it unchanged, and the orchestrator routes it like any other request: aliases,
fallback, breakers, key budgets, the request log and per-second pricing. The voice agents' streaming
speech recognition (ADR 0043) uses it.

## Connect

Open `ws://127.0.0.1:8787/v1/realtime?intent=transcription&model=venom-stt-stream` with your client key in
`Authorization: Bearer <key>`. A browser-style client can send the key as the subprotocol
`openai-insecure-api-key.<key>` instead. Keys never go in the URL: a `key` or `api_key` query parameter is
refused. Leave `model` out and the orchestrator routes on the model of your first `session.update`.

The server sends `session.created` at once. Your first message must be `session.update`, within 5 seconds:

```json
{"type": "session.update",
 "session": {"type": "transcription",
             "audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000},
                                 "transcription": {"model": "venom-stt-stream", "language": "en"},
                                 "turn_detection": null}}},
 "venom": {"partials": "replace"}}
```

Formats: `audio/pcm` (16-bit little-endian mono at 8000, 16000 or 24000 Hz), `audio/pcmu` and `audio/pcma`
(G.711, 8 kHz, as phone lines carry it). The orchestrator converts to whatever the backend takes. Once the
backend's session is up you get `session.updated`, with `venom.upstream` and `venom.model` naming the target.

## Stream

| You send | What it does |
|---|---|
| `input_audio_buffer.append` | `{"audio": "<base64>"}`, 20 to 100 ms per message, at most 1 MiB |
| `input_audio_buffer.commit` | the end of the caller's turn: the backend finalizes the item |
| `input_audio_buffer.clear` | drops the audio not yet committed |

| You get | Meaning |
|---|---|
| `input_audio_buffer.committed` | an item ended (your commit, or the backend's own VAD) |
| `conversation.item.input_audio_transcription.delta` | new text for the item; deltas only ever append |
| `venom.transcription.partial` | with `venom.partials: "replace"`: the whole current guess, and `stable_len`, the part that won't change |
| `conversation.item.input_audio_transcription.completed` | the item's final transcript |
| `input_audio_buffer.speech_started` / `speech_stopped` | from backends that run VAD, with `server_vad` |
| `error` | `{type, code, message, retryable}` |

Some backends revise their partial results (FunASR's 2pass mode first guesses, then corrects each
sentence). An OpenAI-only client gets deltas for the corrected text only; ask for venom partials to see the
guesses too.

## Failures

The orchestrator falls back to the next target only while the session starts. After `session.updated`, an
upstream failure sends an `error` with `retryable: true` and closes with code 1011, and your client
reconnects. 1013 means no upstream could take the session; 4408 means no `session.update` arrived in time;
4400 means a bad first message or model.

Audio is never held back: each frame goes to the backend as it arrives. If the backend stops reading, the
orchestrator drops audio beyond one second rather than queueing it, so a slow backend never stalls your
client.

## Backends

| Kind | Backend |
|---|---|
| `funasr` | FunASR's `funasr-wss-server-2pass`, local, no key (`base_url: ws://host:10095`) |
| `openai` | OpenAI's realtime transcription: `gpt-live-transcribe`, `gpt-transcribe`, `gpt-realtime-whisper` (and the older `gpt-4o-transcribe`, `whisper-1`) |
| `whisperlive` | a WhisperLive server, local, no key (`base_url: ws://host:9090`) |
| `elevenlabs` | ElevenLabs `scribe_v2_realtime`; phone audio passes through as µ-law |
| `dashscope` | Alibaba Model Studio: `qwen3-asr-flash-realtime` (Singapore and Beijing), `fun-asr-realtime`, `paraformer-realtime-v2` / `-8k-v2` (Beijing keys only) |

For a local FunASR server, `docker compose --profile asr up -d funasr` starts one on 127.0.0.1:10095 (it
downloads its models from ModelScope the first time). docs/configuration.md has the upstream example. Each session is one row in the request log, with the audio
in `audio_ms` and its cost at the model's `per_second`. Venom Cloud doesn't relay WebSockets yet: point the
client at a local orchestrator or a daemon it can reach.
