Skip to content
Docs menu

Configuration reference

Every top-level section of orchestrator.yaml with its keys and defaults, plus how the file is loaded, reloaded, versioned and rolled back.

View .md

The orchestrator reads one YAML file, $VENOM_CONFIG (default config/orchestrator.yaml). The shipped file is commented line by line and is the best starting point; this page lists every key.

Loading and reloading

  • ${VAR} and ${VAR:-default} are expanded from the environment before parsing. An unset or empty variable with no default becomes ""; a bare $ is left alone.
  • Parsing is strict: an unknown key, kind or strategy, a duplicate name, or a route target on an unknown upstream stops the load with the field named. Availability is lenient: an upstream whose key variable is empty is kept but disabled.
  • Legal but likely-wrong settings (a ttft_timeout not shorter than defaults.retry_budget, a profile target outside its own allow list) are logged as warnings at every load.
  • Reload without a restart: POST /admin/v1/reload or SIGHUP. Both also re-read the price table. A reload that fails keeps the running config.
curl -s -X POST http://127.0.0.1:8787/admin/v1/reload -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"

Top-level sections

Key Page
listen below
upstreams Upstreams and kinds
routes Routes and strategies
profiles, side_requests Profiles, side requests and gears
gears, gears_locked, gears_source Profiles, side requests and gears
defaults, response_cache, dedup, rate_shaping, batch_routing, pricing, models, guardrails below
search Web search and MCP

listen

listen: ${VENOM_LISTEN:-127.0.0.1:8787} in the shipped file; 127.0.0.1:8787 when absent. Loopback by default: with no client key the gateway serves anyone who can reach it, so it refuses to start on another address without a key unless VENOM_ALLOW_OPEN=1. The Docker image sets VENOM_LISTEN=0.0.0.0:8787 inside the container and compose publishes the port on the host's loopback only.

defaults

Key Default What it does
defaults.timeout 120s Per-attempt time to first byte. Once a byte arrives the request runs until it ends or the client leaves.
defaults.max_inflight 0 (unlimited) Concurrent requests per upstream, unless the upstream sets its own.
defaults.health_interval 15s Health-check period per upstream.
defaults.max_attempts 3 Total attempts per request. Every target still gets one try when there are more targets; extra attempts retry the same targets with backoff.
defaults.retry_budget 20s No new attempt starts after this much time.
defaults.backoff_base, defaults.backoff_max 100ms, 2s Full-jitter exponential backoff. backoff_max is also the longest upstream Retry-After waited on; a longer one goes to the client.
defaults.retry_ratio, defaults.retry_min_per_second 0.2, 1 Per-upstream retry budget: retries are allowed while retries ≤ ratio × first attempts + min_per_second × 10 s over the last 10 s. Both 0 = no same-upstream retries.
defaults.breaker_threshold 3 Consecutive failures that open an upstream's circuit breaker.
defaults.breaker_cooldowns [1m, 5m, 30m] Growing cooldown per trip (the last repeats) before a half-open probe.
defaults.breaker_cooldown none One fixed cooldown instead of the ladder.
defaults.max_body_bytes 33554432 (32 MiB) Request body cap (413 request_too_large). Also caps audio uploads.
defaults.write_stall_timeout 60s A client that takes no bytes for this long during one write is dropped and its upstream request cancelled.
defaults.stream_idle_timeout 5m An upstream that sends nothing for this long mid-response is cut off; the stream ends with its error event. Raise it for models that think for minutes without a byte.
defaults.request_log_retention 100000 Request-log rows kept in SQLite (the spend ledger is never pruned).
defaults.load_interval 2s How often vllm, sglang and llamacpp engines' /metrics are read for queue and KV-cache use.
defaults.saturation_kv, defaults.saturation_waiting 0.95, 4 When an engine counts as saturated (KV cache that full, or that many requests queued); a saturated engine goes behind the others.

response_cache

Exact-match reuse of deterministic side requests (a side kind at temperature 0, or any kind listed in kinds) from the same client key; never a main-session turn. A hit is billed $0 and answered with x-venom-response-cache: hit, or shared when an identical request was in flight.

Key Default What it does
response_cache.ttl 5m How long a response is reused.
response_cache.max_bytes 33554432 (32 MiB) Memory bound, least recently used evicted; negative turns the cache off.
response_cache.max_entry_bytes 1048576 (1 MiB) Largest response stored.
response_cache.kinds [] (shipped: [auto-classifier, decide]) Side kinds cached at any temperature.

dedup

Identical concurrent requests from one key (temperature 0 set explicitly, /v1/embeddings, /v1/rerank, or any request with the same Idempotency-Key) share one upstream call, fanned out as it streams. A request opts out with x-venom-dedup: off. Duplicates are billed once and answered with x-venom-dedup: joined; a completed Idempotency-Key response is replayed with x-venom-dedup: replayed and Idempotent-Replayed: true. The same key with another body is 422 idempotency_key_reused; a key over 255 bytes is 400 invalid_idempotency_key.

Key Default What it does
dedup.max_bytes 1048576 (1 MiB) A duplicate may join while the response is shorter; a longer Idempotency-Key response is not kept.
dedup.max_lag_bytes 262144 (256 KiB) A reader that falls this far behind is cut loose, so a slow client never holds the others.
dedup.idempotency_ttl 5m How long a completed Idempotency-Key response is replayed.
dedup.idempotency_max_bytes 33554432 (32 MiB) Bound on kept responses; -1 keeps none (collapsed only while in flight).

rate_shaping

Each upstream learns its provider's rate limits from x-ratelimit-* and anthropic-ratelimit-* headers and from 429s. A request that would exceed a learned limit goes to a later target with room, else waits in that upstream's queue in fair order across client keys. A request that waited carries x-venom-queued-ms. Only a full queue or a wait past the bound answers 429 upstream_queue_full or upstream_wait_exceeded, with Retry-After. Upstreams that never send rate-limit headers (local engines) are not shaped.

Key Default What it does
rate_shaping.max_queue 256 Requests waiting per upstream; negative = none wait.
rate_shaping.max_wait 30s Longest wait; negative = none wait (limits then only steer requests).
rate_shaping.weights each key 1 Fair-queuing weight by client key name (at most 1000).

batch_routing

Shapes batch requests (x-venom-urgency: batch); see Batches, files and service tiers.

Key Default What it does
batch_routing.window 30s (at most 24h) How long requests for one model collect before their provider batch is submitted.
batch_routing.max_lines 1000 (at most 50,000) Submit at once at this many requests.
batch_routing.max_wait 10m (at most 1h) Cap on x-venom-batch-wait and GET /v1/batch_requests/{id}?wait=.
batch_routing.callback_hosts [] (no callbacks) Hosts (host or host:port) x-venom-callback-url may name. The orchestrator calls no client-named URL unless you list its host.

pricing

Costs are list prices from the offline price table (prices.yaml next to the config, or $VENOM_PRICES; never fetched at runtime): USD per 1M input, cached_input and output tokens, with optional cache_write, flex and priority rates. Local upstreams cost $0. A cloud model nothing prices costs $0 and is logged once; GET /admin/v1/pricing lists every model's effective price and source.

pricing:
  source: [custom, builtin]
  models:
    azure/my-gpt: {base_model: gpt-5.4}                  # priced as gpt-5.4
    my-tuned:     {input: 3, cached_input: 0.3, output: 12}
    tts-hosted/narrator: {per_1m_chars: 15}              # text to speech, per 1M characters
Key Default What it does
pricing.source [custom, builtin] Lookup order over your prices (custom) and the table (builtin); a source left out is not consulted.
pricing.models none Rules keyed by upstream/model (that upstream only) or a bare model id (any upstream).
pricing.models.*.base_model none The model whose price applies. A served model with a base model is answered with x-venom-base-model.
pricing.models.*.input, cached_input, output none Your price in USD per 1M tokens; input and output together, cached_input defaults to input. Wins over base_model.
pricing.models.*.search_unit none USD per search unit, for a reranker billed per search.
pricing.models.*.per_1m_chars none USD per 1M input characters, for text-to-speech.
pricing.models.*.per_second none USD per second of audio, for speech-to-text.

The same pricing keys are accepted in an upstream's model_info and on a route target; one model priced two different ways is a config error.

models

Per-model settings a mesh daemon advertises; today only provenance for the public mesh.

Key What it does
models[].id A model id the daemon serves (a route alias or upstream/model). Unique.
models[].upstream The vllm or sglang upstream that runs it; public jobs go to it alone. Required with provenance.
models[].provenance.registry_id The public mesh registry entry it runs.
models[].provenance.sampler Must be venom-sampler/1.

guardrails

guardrails: holds the layered guardrails checked on every /v1 inference request: the org's rules and catalog picks, teams: keyed by a client key's team label and members: keyed by a key's name. A block is 403 guardrail_blocked with a code such as GR-ORG-CRED-101; a redaction, warning or keep-local match passes with an x-venom-guardrail header per rule. GET /v1/guardrails shows a key what applies to it, and GET /g/{code} (no key) is a code's page. Absent, nothing is checked and nothing costs. The rule format, codes and layers are in the Security section.

Config history and rollback

Every config the process applies (start, reload, SIGHUP, a local-model selection, a rollback) is stored in SQLite as the file's bytes before ${VAR} expansion, with its SHA-256; the newest 50 are kept.

curl -s http://127.0.0.1:8787/admin/v1/config/versions -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"
curl -s "http://127.0.0.1:8787/admin/v1/config/diff?from=12&to=file" -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"
curl -s -X POST http://127.0.0.1:8787/admin/v1/config/versions/12/rollback \
  -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -H "Content-Type: application/json" -d '{}'
  • GET /admin/v1/config/versions lists versions newest first, marks the current one and says whether the file on disk differs (file.changed).
  • GET /admin/v1/config/diff?from=<id|file>&to=<id|file> diffs two versions (default: the newest and the one before) by upstream, route, profile and other key. Secrets are redacted: ${VAR} stays as written, literal header values and secret-named fields become a short hash.
  • POST /admin/v1/config/versions/{id}/rollback writes that version to the file and reloads it. 409 config_changed_on_disk when the file isn't what was last applied (send overwrite_sha256 equal to the reported disk_sha256 to replace it anyway), 422 invalid_version when it no longer loads, 500 config_not_writable for a read-only mount, 422 reload_failed restores the previous file.