Skip to content
Docs menu

Upstreams and kinds

Add an inference server or cloud provider to the orchestrator, pick its kind, and set its keys, models, capabilities and limits.

View .md

An upstream is one server the orchestrator sends requests to: a local engine, a provider's API or a gateway. Each entry under upstreams: in orchestrator.yaml has a name (what routes and upstream/model ids refer to), a kind (which wire format and parameter rules apply) and a base_url. Routes then list {upstream, model} targets; see Routes and strategies.

Minimal example

upstreams:
  - name: ollama
    kind: ollama
    base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1}
    context_window: ${OLLAMA_CONTEXT_LENGTH:-32768}
  - name: openai
    kind: openai
    base_url: https://api.openai.com/v1
    api_key_env: OPENAI_API_KEY      # unset = the upstream is disabled, nothing calls OpenAI
    max_inflight: 32

routes:
  - alias: venom-default
    targets:
      - {upstream: ollama, model: "qwen3:8b"}
      - {upstream: openai, model: gpt-5.4-mini}

An upstream whose api_key_env names an empty variable, or whose base_url is empty, is kept but disabled: it is never health-checked and its targets are skipped. That is why one file works online and off the grid. GET /admin/v1/upstreams shows each upstream's state and why it is disabled.

Kinds

The kind decides the request rewriting ("quirks"), the capability defaults, how the context window is discovered and how the key is sent. Unset, kind is generic.

Kind For Notes
openai OpenAI's API /v1/responses passed through natively; reasoning_effort mapped to the nearest level each model takes; service_tier flex and priority; a shared prompt_cache_key pool per model; native_batch and cache_retention (24h, 30m) apply.
ollama Ollama's OpenAI-compatible API max_completion_tokens sent as max_tokens; n and logit_bias dropped; tool_choice off by default; reasoning read from reasoning; context window from /api/ps (loaded model) or /api/show; a model without the tools capability is routed around. Local.
vllm vLLM stream_options only on streams, an empty tools dropped; reasoning in reasoning; window from /v1/models max_model_len; load scraped from /metrics. Local.
sglang SGLang reasoning_content; window from /v1/models max_model_len; /metrics scraped. Local.
llamacpp llama.cpp's llama-server cache_prompt: true; a slot per x-venom-session (id_slot) on builds without the host prompt cache; window from /v1/models meta.n_ctx or /props. slots applies. Local.
generic any other OpenAI-compatible server (LiteLLM, LocalAI, Kimi's platform, Azure OpenAI's v1 API, Together, Speaches, Kokoro-FastAPI) No rewriting beyond the common rules; cache_retention: 1h is Kimi's.
zai Z.ai's GLM API (https://api.z.ai/api/paas/v4, or …/coding/paas/v4 for the GLM Coding Plan) temperature clamped to [0, 1], top_p to [0.01, 1], stop as an array, metadata dropped, GLM-5.3 efforts mapped to the ones it takes; tool_choice off. Context windows come from config.
dashscope Alibaba Cloud Model Studio's OpenAI-compatible API (Qwen, DeepSeek, Kimi and GLM hosted there) Thinking controls (enable_thinking, thinking_budget) set from reasoning_effort per model family; a forced tool choice turns thinking off; explicit_cache and native_batch apply; rerank defaults to Model Studio's native API.
mistral Mistral AI's La Plateforme (https://api.mistral.ai/v1) Only the documented fields are sent (the API rejects others with 422); seed → random_seed; tool call ids shortened to 9 characters; /v1/completions with a suffix goes to Codestral's /fim/completions; embeddings dimensions → output_dimension. Window from /v1/models max_context_length.
minimax MiniMax's OpenAI-compatible API (https://api.minimax.io/v1, or https://api.minimaxi.com/v1 in mainland China) temperature and top_p kept in (0, 1]; reasoning_split on; effort and thinking knobs per model; tool_choice off. No /models: list the models. Chat only.
anthropic Anthropic's Messages API (https://api.anthropic.com/v1) Chat completions and /v1/responses translated to POST /messages and back; x-api-key auth; prompt-cache breakpoints placed for you; cache_retention: 1h and native_batch apply; embeddings and completions are unsupported_endpoint.
bedrock Amazon Bedrock's Converse API Chat translated to /model/{id}/converse(-stream); SigV4-signed with IAM keys (aws:); embeddings and rerank through InvokeModel; priced by the model's base name (us.anthropic.claude-sonnet-5-5-v1:0 as claude-sonnet-5-5). models is required.
sagemaker an AWS SageMaker real-time endpoint Models are endpoint names (endpoint:component sets the inference component); payload: openai or tgi; SigV4. models is required, no health probe.
typesafe TypeSafe's System One decision API Decision models: they serve POST /v1/systemone only. See Rerank, images and decisions.
sdapi AUTOMATIC1111's web UI API (also Forge, SD.Next, stable-diffusion.cpp's server); base_url is the server root without /v1 Image models: they serve POST /v1/images/generations only.

"Local" kinds (ollama, vllm, sglang, llamacpp), and any upstream whose base URL is loopback, private, link-local, *.local, *.internal or a single-label host, cost $0 and are never counted against budgets. Set local: false for a paid gateway on a private address, local: true for your own server on a public name.

Speech providers with their own wire formats register as kinds of their own (ADR 0044): cloud ones (elevenlabs, fishaudio, byteplus-speech) and local servers (cosyvoice, qwen3-tts, indextts, sparktts, fishspeech, which cost $0 wherever they run). Speech also goes to any kind above that speaks OpenAI's audio API. See the Voice section and Local speech models.

Upstream keys

Key Type Default What it does
name string required Unique, no /.
kind string generic One of the kinds above.
base_url URL none (disabled) The API root including /v1 (except sdapi). AWS kinds default to the regional endpoint.
api_key_env env var name none (no auth) The variable holding the key; unset or empty disables the upstream.
api_key_envs list of env var names none A credential chain after api_key_env: a request refused for credit (402, insufficient_quota), a revoked key (401, 403) or a rate limit (429) is retried on the next key before the next route target.
models list discovered The models it serves. Required for bedrock, sagemaker, minimax; when set it is also an allowlist for upstream/<id> and bare ids.
timeout duration defaults.timeout (120s) Per-attempt time to first byte. A started response runs to its end.
max_inflight int defaults.max_inflight (0 = unlimited) Concurrent requests to this upstream; /v1/models reports it so clients queue on their side.
headers map none Extra request headers (values may use ${VAR}).
health_interval duration defaults.health_interval (15s) How often it is probed.
stream_usage bool by kind Whether stream_options.include_usage is injected so streams report usage.
context_window int (tokens) discovered, else unknown Applies to every model unless model_info overrides it. A request too long for every target is a 400 before any attempt.
capabilities map by kind See below; applies to every model.
model_info map of model id → settings none Per-model context_window, capabilities, and pricing (base_model, input, cached_input, output, or a nested price with input, cached_input, output, search_unit).
slots int /props total_slots llamacpp only: the slot count sessions are pinned onto.
metrics_url URL origin + /metrics for vllm, sglang, llamacpp Where the engine's load (queue, KV-cache use) is scraped.
local bool by kind and address Whether its tokens cost $0.
native_batch bool false openai, dashscope, anthropic only: hand batches and batch requests to the provider's Batch API at its batch price. See Batches.
aws map from the environment bedrock and sagemaker: region (default AWS_REGION, then AWS_DEFAULT_REGION), access_key_id_env (default AWS_ACCESS_KEY_ID), secret_access_key_env (default AWS_SECRET_ACCESS_KEY), session_token_env (default AWS_SESSION_TOKEN). Disabled while the region or a key is missing.
payload openai or tgi openai sagemaker only: the container's body format.
prompt_template plain, chatml, llama3 plain sagemaker with payload: tgi: how messages become one prompt.
rerank jina, cohere, voyage, tei, dashscope dashscope for kind dashscope, InvokeModel for bedrock, else jina The dialect POST /v1/rerank is sent in.
reference_audio_dir path none Local speech kinds: the directory a speech request's venom_options.reference_audio file path is read from for voice cloning; unset, file paths are refused. See Local speech models.

Capabilities

capabilities: on an upstream, or under model_info.<model>.capabilities, overrides the kind's defaults. Discovery (after each passing health check) only fills what the config leaves unset.

Key Type Default What it does
tools bool true (false for typesafe) The model takes tools; a request with tools skips targets that don't.
tool_choice bool true; false for ollama, zai, minimax Whether tool_choice is sent or removed.
responses bool true for openai Native /v1/responses; otherwise requests are translated to chat completions.
reasoning_field reasoning or reasoning_content reasoning_content (reasoning on ollama, vllm; none on openai) Where the server puts reasoning text; answers are normalised to reasoning_content.
replay_reasoning bool true (false for openai, anthropic, bedrock) Whether earlier turns' reasoning is sent back.
vision bool discovered, else a known-model list Image input; /v1/models reports it.
decisions bool true for typesafe A decision model: serves /v1/systemone and nothing else.
tts bool a known-id list (tts-1, gpt-4o-mini-tts, Kokoro, ...) A text-to-speech model for /v1/audio/speech.
stt bool a known-id list (whisper, *-transcribe, voxtral, ...) A speech-to-text model; reported by /v1/models.
constrained_decoding bool true for local kinds The server decodes against a JSON schema; false removes json_schema constraints instead of sending them.
cache_retention 24h, 30m (openai); 1h (generic, anthropic) none The longer prompt-cache lifetime asked for when a session pauses (x-venom-cache-ttl: long, or gaps over 5 minutes).
explicit_cache bool off dashscope only: Model Studio's explicit cache markers on prompts of 1,024+ tokens.

Examples

A paid gateway on a private address, priced and budgeted:

- name: litellm
  kind: generic
  base_url: http://10.0.0.5:4000/v1
  api_key_env: LITELLM_KEY
  local: false

Two keys for one provider, the second used when the first runs out of credit:

- name: openai
  kind: openai
  base_url: https://api.openai.com/v1
  api_key_env: OPENAI_API_KEY
  api_key_envs: [OPENAI_API_KEY_BACKUP]

Bedrock and a SageMaker endpoint priced per token:

- name: bedrock
  kind: bedrock
  aws: {region: us-east-1}
  models: ["us.anthropic.claude-sonnet-5-5-v1:0"]
- name: sagemaker
  kind: sagemaker
  aws: {region: us-east-1}
  payload: openai
  models: [my-llama-endpoint]
  model_info:
    my-llama-endpoint: {context_window: 131072, price: {input: 0.30, output: 0.60}}

Mistral and MiniMax (both commented out in the shipped config; set MISTRAL_API_KEY or MINIMAX_API_KEY and uncomment):

- name: mistral
  kind: mistral
  base_url: ${MISTRAL_BASE_URL:-https://api.mistral.ai/v1}
  api_key_env: MISTRAL_API_KEY
  models: [mistral-large-latest, mistral-small-latest, codestral-latest, mistral-embed]
- name: minimax
  kind: minimax
  base_url: ${MINIMAX_BASE_URL:-https://api.minimax.io/v1}
  api_key_env: MINIMAX_API_KEY
  models: [MiniMax-M3, MiniMax-M3.1-Flash]

Troubleshooting

  • The upstream shows disabled: its api_key_env variable is empty in the orchestrator's environment, or base_url is empty. Set it and POST /admin/v1/reload (or send SIGHUP).
  • 503 model_unavailable: the upstream's own model list (Ollama, vLLM) lacks the model. The error names the fix, such as ollama pull qwen3:8b.
  • A target is skipped as unhealthy: its last health check failed. Kinds without a model list (minimax, AWS) are judged by their circuit breaker instead.
  • 502 upstream_auth_failed: the provider refused the orchestrator's key (not the client's). The provider's body is not relayed, since it can echo part of the key.
  • Config errors at start or reload: unknown fields and kinds, a duplicate name, a / in a name, or a kind-specific key on the wrong kind (slots off llamacpp, native_batch off the three batch kinds, payload off sagemaker) stop the load with the field named. A failed reload keeps the running config.