Skip to content
Docs menu

Orchestrator

Run venom's self-hosted OpenAI-compatible gateway in front of your own inference servers and cloud providers, and call it from any OpenAI SDK.

View .md

The orchestrator is a single Go binary that speaks the OpenAI HTTP API. It sits in front of every inference backend you run (vLLM, SGLang, llama.cpp, Ollama, image servers, speech servers) and any cloud provider you hold a key for, and gives clients one endpoint with model aliases, fallback before the first byte, circuit breakers, session-sticky routing for prompt-cache reuse, per-key limits and budgets, and a request log with cost, all in SQLite. Streams are relayed as they arrive and never buffered.

It is what the venom CLI talks to by default (VENOM_BASE_URL=http://localhost:8787/v1), what the desktop daemon runs, and what a Venom Cloud daemon is in mesh mode. Any program that uses an OpenAI SDK can use it unchanged.

Start one

From the release binaries (the venom-orchestrator-<version>-<os>-<arch> file in the public releases repository, with orchestrator.yaml and prices.yaml from the same release next to it):

export VENOM_CONFIG=./orchestrator.yaml   # default: config/orchestrator.yaml
export VENOM_API_KEY=my-local-key         # the bootstrap client key
export VENOM_ADMIN_TOKEN=my-admin-token   # turns the admin API on
./venom-orchestrator

With Docker (the release builds the venom/orchestrator:<version> image with the shipped config inside):

docker run -e VENOM_API_KEY=my-local-key -p 127.0.0.1:8787:8787 venom/orchestrator:<version>

From a checkout of the repository, make up SERVICES=orchestrator runs it in Docker, make up SERVICES="orchestrator ollama" adds a local Ollama, and make dev-orchestrator runs it on the host with hot reload.

The shipped config routes venom-default to a local vLLM, then a local Ollama, then OpenAI. Any upstream whose key variable is empty is disabled, so with no cloud keys the same file runs fully local. Pull a model and ask:

ollama pull qwen3:8b
curl -s http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer my-local-key" -H "Content-Type: application/json" \
  -d '{"model":"venom-default","messages":[{"role":"user","content":"Say hi"}]}'

From Python or TypeScript, point the OpenAI SDK at it:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="my-local-key")
reply = client.chat.completions.create(
    model="venom-default", messages=[{"role": "user", "content": "Say hi"}]
)
print(reply.choices[0].message.content)
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://127.0.0.1:8787/v1", apiKey: "my-local-key" });
const reply = await client.chat.completions.create({
  model: "venom-default",
  messages: [{ role: "user", content: "Say hi" }],
});
console.log(reply.choices[0].message.content);

The answer headers say what served it: x-venom-upstream, x-venom-model and, when the first target did not, x-venom-fallback with the reason.

Authentication

Surface Credential When it is missing
/v1/* and /mcp Authorization: Bearer <client key>: the bootstrap key in VENOM_API_KEY, or a vk-… key minted with POST /admin/v1/keys With no client key at all the orchestrator runs in open mode and serves anyone who can reach it. It refuses to start that way on a non-loopback listen address unless VENOM_ALLOW_OPEN=1.
/admin/v1/* Authorization: Bearer $VENOM_ADMIN_TOKEN The admin API answers 503 ("admin API disabled"). A wrong token is 401.
/healthz, /readyz, /metrics, /g/{code}, /api/models/providers/{id} none

A client key can be limited to some models (allowed_models), given a request and token rate (rpm_limit, tpm_limit), a daily and monthly spend budget, a team label and a default route profile. See Keys, budgets and limits.

What it serves

Area Endpoints Page
Models GET /v1/models, GET /v1/models/{id}, GET /v1/gears, POST /v1/prewarm Chat, responses and models, Profiles and gears
Text POST /v1/chat/completions, POST /v1/completions, POST /v1/responses (and background responses), POST /v1/embeddings Chat, responses and models
Retrieval POST /v1/rerank, POST /v1/search, GET /v1/search/backends, /mcp Rerank, images and decisions, Web search and MCP
Media POST /v1/audio/transcriptions, POST /v1/audio/translations, POST /v1/audio/speech, GET /v1/audio/voices, POST /v1/images/generations Voice, Rerank, images and decisions
Decisions POST /v1/systemone Rerank, images and decisions
Offline work /v1/files, /v1/batches, /v1/batch_requests Batches, files and service tiers
Your own figures GET /v1/usage, GET /v1/routing, GET /v1/guardrails Keys, budgets and limits
Operators /admin/v1/* Admin API
Probes /healthz, /readyz, /metrics Operations

Any other path under /v1/ answers 404 with code unknown_url, and a known path with the wrong method answers 405 method_not_allowed with an Allow header. The full endpoint list is generated from the contract in the API reference (/docs/api/orchestrator).

How a request is routed

A request's model is resolved in this order: a route alias (its target list, or for an experiment the variant this session is assigned); upstream/model (that upstream directly); a bare model id (the first upstream that lists it). An id with a / whose prefix upstream can't serve it is looked up as a bare id, so vLLM's openai/gpt-oss-20b still reaches vLLM.

The targets are tried in the route's strategy order. Connection errors, 429, 5xx and a missed time-to-first-token deadline move to the next target, but only before the first byte reaches the client; after that a failure ends the stream with an error event. Routes and strategies has the rules.

Errors

Errors use OpenAI's shape, {"error": {"message", "type", "code"}}. They name the upstream but never its address or the transport error. Codes you will meet most:

Status Code Meaning
400 context_length_exceeded no target's context window holds the request (prompt bytes/4 plus its max output); nothing is truncated
400 unknown_profile the x-venom-profile header names no configured profile
401 missing or wrong client key
403 model_not_allowed, profile_forbidden, unpriced_model, guardrail_blocked the key, profile, budget or a guardrail refuses it
404 model_not_found no alias, upstream or model by that name (or not served under the profile)
408 request_timeout the body was not read within 60 s
413 request_too_large the body is over defaults.max_body_bytes
429 rate_limit_exceeded, tpm_limit_exceeded, budget_exceeded, upstream_queue_full, upstream_wait_exceeded a limit or budget; see Keys, budgets and limits
502 upstream_auth_failed, upstream_error every target failed; the orchestrator's own credential was refused
503 no_upstream, model_unavailable no target is enabled, or every target lacks its model (each named with its fix)

In this section

  • Upstreams and kinds: Add an inference server or cloud provider to the orchestrator, pick its kind, and set its keys, models, capabilities and limits.
  • Provider keys and setup: Add API keys for cloud providers, local engine URLs and voice providers to a self-hosted orchestrator without editing .env.
  • Routes and strategies: Define model aliases, choose how their targets are ordered, and understand fallback, retries, stickiness, experiments and shadow traffic.
  • Profiles, side requests and gears: Re-target aliases per request with route profiles, give side requests their own body defaults, and publish gears that clients list.
  • Configuration reference: Every top-level section of orchestrator.yaml with its keys and defaults, plus how the file is loaded, reloaded, versioned and rolled back.
  • Orchestrator environment variables: Every environment variable the orchestrator, its shipped config, the desktop daemon and the public-mesh auditor read, with defaults.
  • Chat, responses, embeddings and models: Call chat completions, completions, the Responses API (including background responses) and embeddings through the orchestrator, and read what the model list tells you.
  • Keys, budgets and limits: Mint client keys, limit what each may call and spend, cap intents and teams, and let each key read its own usage and routing.
  • Batches, files and service tiers: Run offline work with OpenAI's Batch API, turn single requests into batch requests at batch prices, upload files, and ask for flex or priority tiers.
  • Rerank, images and decisions: Rerank documents, generate images and ask typed decision questions through the orchestrator's routing, keys and budgets.
  • Web search and MCP: Give agents web search over backends you choose, with the search keys kept on the server, through POST /v1/search or as an MCP server for other tools.
  • Admin API: Read the orchestrator's live state, request log, sessions, spend, cache and pricing reports, and manage keys and config over /admin/v1.
  • Operations - probes, metrics, local models and mesh: Monitor the orchestrator with health probes, Prometheus metrics and the profiler, manage local models, connect it to Venom Cloud, and replay traffic from cassettes.