---
title: Orchestrator
description: Run venom's self-hosted OpenAI-compatible gateway in front of your own inference servers and cloud providers, and call it from any OpenAI SDK.
section: orchestrator
order: 0
audience: [operator, builder, agent]
sources: [services/go-orchestrator/AGENTS.md, docs/architecture.md, config/orchestrator.yaml, services/go-orchestrator/internal/server/server.go]
features: [orchestrator.api-root]
---

The orchestrator is a single Go binary that speaks the OpenAI HTTP API. It sits in front of every inference backend you run (vLLM, SGLang, llama.cpp, Ollama, image servers, speech servers) and any cloud provider you hold a key for, and gives clients one endpoint with model aliases, fallback before the first byte, circuit breakers, session-sticky routing for prompt-cache reuse, per-key limits and budgets, and a request log with cost, all in SQLite. Streams are relayed as they arrive and never buffered.

It is what the venom CLI talks to by default (`VENOM_BASE_URL=http://localhost:8787/v1`), what the desktop daemon runs, and what a Venom Cloud daemon is in mesh mode. Any program that uses an OpenAI SDK can use it unchanged.

## Start one {#quickstart}

From the release binaries (the `venom-orchestrator-<version>-<os>-<arch>` file in the [public releases repository](https://github.com/elishahbiz/venom-releases/releases/latest), with `orchestrator.yaml` and `prices.yaml` from the same release next to it):

```bash
export VENOM_CONFIG=./orchestrator.yaml   # default: config/orchestrator.yaml
export VENOM_API_KEY=my-local-key         # the bootstrap client key
export VENOM_ADMIN_TOKEN=my-admin-token   # turns the admin API on
./venom-orchestrator
```

With Docker (the release builds the `venom/orchestrator:<version>` image with the shipped config inside):

```bash
docker run -e VENOM_API_KEY=my-local-key -p 127.0.0.1:8787:8787 venom/orchestrator:<version>
```

From a checkout of the repository, `make up SERVICES=orchestrator` runs it in Docker, `make up SERVICES="orchestrator ollama"` adds a local Ollama, and `make dev-orchestrator` runs it on the host with hot reload.

The shipped config routes `venom-default` to a local vLLM, then a local Ollama, then OpenAI. Any upstream whose key variable is empty is disabled, so with no cloud keys the same file runs fully local. Pull a model and ask:

```bash
ollama pull qwen3:8b
curl -s http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer my-local-key" -H "Content-Type: application/json" \
  -d '{"model":"venom-default","messages":[{"role":"user","content":"Say hi"}]}'
```

From Python or TypeScript, point the OpenAI SDK at it:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="my-local-key")
reply = client.chat.completions.create(
    model="venom-default", messages=[{"role": "user", "content": "Say hi"}]
)
print(reply.choices[0].message.content)
```

```ts
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://127.0.0.1:8787/v1", apiKey: "my-local-key" });
const reply = await client.chat.completions.create({
  model: "venom-default",
  messages: [{ role: "user", content: "Say hi" }],
});
console.log(reply.choices[0].message.content);
```

The answer headers say what served it: `x-venom-upstream`, `x-venom-model` and, when the first target did not, `x-venom-fallback` with the reason.

## Authentication {#auth}

| Surface | Credential | When it is missing |
|---|---|---|
| `/v1/*` and `/mcp` | `Authorization: Bearer <client key>`: the bootstrap key in `VENOM_API_KEY`, or a `vk-…` key minted with `POST /admin/v1/keys` | With no client key at all the orchestrator runs in open mode and serves anyone who can reach it. It refuses to start that way on a non-loopback listen address unless `VENOM_ALLOW_OPEN=1`. |
| `/admin/v1/*` | `Authorization: Bearer $VENOM_ADMIN_TOKEN` | The admin API answers `503` ("admin API disabled"). A wrong token is `401`. |
| `/healthz`, `/readyz`, `/metrics`, `/g/{code}`, `/api/models/providers/{id}` | none | |

A client key can be limited to some models (`allowed_models`), given a request and token rate (`rpm_limit`, `tpm_limit`), a daily and monthly spend budget, a team label and a default route profile. See [Keys, budgets and limits](/docs/orchestrator/keys-and-budgets.md).

## What it serves {#endpoints}

| Area | Endpoints | Page |
|---|---|---|
| Models | `GET /v1/models`, `GET /v1/models/{id}`, `GET /v1/gears`, `POST /v1/prewarm` | [Chat, responses and models](/docs/orchestrator/chat-and-responses.md), [Profiles and gears](/docs/orchestrator/profiles-and-gears.md) |
| Text | `POST /v1/chat/completions`, `POST /v1/completions`, `POST /v1/responses` (and background responses), `POST /v1/embeddings` | [Chat, responses and models](/docs/orchestrator/chat-and-responses.md) |
| Retrieval | `POST /v1/rerank`, `POST /v1/search`, `GET /v1/search/backends`, `/mcp` | [Rerank, images and decisions](/docs/orchestrator/rerank-images-decisions.md), [Web search and MCP](/docs/orchestrator/search-and-mcp.md) |
| Media | `POST /v1/audio/transcriptions`, `POST /v1/audio/translations`, `POST /v1/audio/speech`, `GET /v1/audio/voices`, `POST /v1/images/generations` | [Voice](/docs/voice.md), [Rerank, images and decisions](/docs/orchestrator/rerank-images-decisions.md) |
| Decisions | `POST /v1/systemone` | [Rerank, images and decisions](/docs/orchestrator/rerank-images-decisions.md) |
| Offline work | `/v1/files`, `/v1/batches`, `/v1/batch_requests` | [Batches, files and service tiers](/docs/orchestrator/batches-and-files.md) |
| Your own figures | `GET /v1/usage`, `GET /v1/routing`, `GET /v1/guardrails` | [Keys, budgets and limits](/docs/orchestrator/keys-and-budgets.md) |
| Operators | `/admin/v1/*` | [Admin API](/docs/orchestrator/admin-api.md) |
| Probes | `/healthz`, `/readyz`, `/metrics` | [Operations](/docs/orchestrator/operations.md) |

Any other path under `/v1/` answers `404` with code `unknown_url`, and a known path with the wrong method answers `405 method_not_allowed` with an `Allow` header. The full endpoint list is generated from the contract in the API reference (`/docs/api/orchestrator`).

## How a request is routed {#routing}

A request's `model` is resolved in this order: a route alias (its target list, or for an experiment the variant this session is assigned); `upstream/model` (that upstream directly); a bare model id (the first upstream that lists it). An id with a `/` whose prefix upstream can't serve it is looked up as a bare id, so vLLM's `openai/gpt-oss-20b` still reaches vLLM.

The targets are tried in the route's strategy order. Connection errors, `429`, `5xx` and a missed time-to-first-token deadline move to the next target, but only before the first byte reaches the client; after that a failure ends the stream with an error event. [Routes and strategies](/docs/orchestrator/routing.md) has the rules.

## Errors {#errors}

Errors use OpenAI's shape, `{"error": {"message", "type", "code"}}`. They name the upstream but never its address or the transport error. Codes you will meet most:

| Status | Code | Meaning |
|---|---|---|
| 400 | `context_length_exceeded` | no target's context window holds the request (prompt bytes/4 plus its max output); nothing is truncated |
| 400 | `unknown_profile` | the `x-venom-profile` header names no configured profile |
| 401 | | missing or wrong client key |
| 403 | `model_not_allowed`, `profile_forbidden`, `unpriced_model`, `guardrail_blocked` | the key, profile, budget or a guardrail refuses it |
| 404 | `model_not_found` | no alias, upstream or model by that name (or not served under the profile) |
| 408 | `request_timeout` | the body was not read within 60 s |
| 413 | `request_too_large` | the body is over `defaults.max_body_bytes` |
| 429 | `rate_limit_exceeded`, `tpm_limit_exceeded`, `budget_exceeded`, `upstream_queue_full`, `upstream_wait_exceeded` | a limit or budget; see [Keys, budgets and limits](/docs/orchestrator/keys-and-budgets.md) |
| 502 | `upstream_auth_failed`, `upstream_error` | every target failed; the orchestrator's own credential was refused |
| 503 | `no_upstream`, `model_unavailable` | no target is enabled, or every target lacks its model (each named with its fix) |

## Related

- [Upstreams and kinds](/docs/orchestrator/upstreams.md), [Routes and strategies](/docs/orchestrator/routing.md), [Configuration reference](/docs/orchestrator/configuration.md), [Environment variables](/docs/orchestrator/environment.md).
- The venom CLI uses the orchestrator as its provider: [Getting started](/docs/getting-started.md).


## In this section {#in-this-section}

- [Upstreams and kinds](/docs/orchestrator/upstreams.md): Add an inference server or cloud provider to the orchestrator, pick its kind, and set its keys, models, capabilities and limits.
- [Provider keys and setup](/docs/orchestrator/providers-setup.md): Add API keys for cloud providers, local engine URLs and voice providers to a self-hosted orchestrator without editing .env.
- [Routes and strategies](/docs/orchestrator/routing.md): Define model aliases, choose how their targets are ordered, and understand fallback, retries, stickiness, experiments and shadow traffic.
- [Profiles, side requests and gears](/docs/orchestrator/profiles-and-gears.md): Re-target aliases per request with route profiles, give side requests their own body defaults, and publish gears that clients list.
- [Configuration reference](/docs/orchestrator/configuration.md): Every top-level section of orchestrator.yaml with its keys and defaults, plus how the file is loaded, reloaded, versioned and rolled back.
- [Orchestrator environment variables](/docs/orchestrator/environment.md): Every environment variable the orchestrator, its shipped config, the desktop daemon and the public-mesh auditor read, with defaults.
- [Chat, responses, embeddings and models](/docs/orchestrator/chat-and-responses.md): Call chat completions, completions, the Responses API (including background responses) and embeddings through the orchestrator, and read what the model list tells you.
- [Keys, budgets and limits](/docs/orchestrator/keys-and-budgets.md): Mint client keys, limit what each may call and spend, cap intents and teams, and let each key read its own usage and routing.
- [Batches, files and service tiers](/docs/orchestrator/batches-and-files.md): Run offline work with OpenAI's Batch API, turn single requests into batch requests at batch prices, upload files, and ask for flex or priority tiers.
- [Rerank, images and decisions](/docs/orchestrator/rerank-images-decisions.md): Rerank documents, generate images and ask typed decision questions through the orchestrator's routing, keys and budgets.
- [Web search and MCP](/docs/orchestrator/search-and-mcp.md): Give agents web search over backends you choose, with the search keys kept on the server, through POST /v1/search or as an MCP server for other tools.
- [Admin API](/docs/orchestrator/admin-api.md): Read the orchestrator's live state, request log, sessions, spend, cache and pricing reports, and manage keys and config over /admin/v1.
- [Operations - probes, metrics, local models and mesh](/docs/orchestrator/operations.md): Monitor the orchestrator with health probes, Prometheus metrics and the profiler, manage local models, connect it to Venom Cloud, and replay traffic from cassettes.
