Skip to content
Docs menu

Add a provider key and route by cost

Add a cloud provider to your orchestrator next to local models, make an alias that tries the cheapest target first, choose it per key, request or gear, and read back what it cost.

View .md

This recipe starts from an orchestrator serving local models (offline setup or the orchestrator quick start) and adds a cloud provider as a cheaper or stronger option, without sending anything to it until you choose.

1. Add the key

Provider keys live in the orchestrator's environment only, never in the CLI's or in a config file. The shipped config already has upstreams for these, each disabled while its variable is empty:

Variable Upstream (name, kind) Notes
OPENAI_API_KEY openai, openai OPENAI_API_KEY_2 and more as fallback keys with api_key_envs
QWEN_API_KEY qwen, dashscope Alibaba Cloud Model Studio; QWEN_BASE_URL picks the region
ZAI_API_KEY zai, zai Z.ai GLM; ZAI_BASE_URL picks pay-as-you-go or the Coding Plan
MISTRAL_API_KEY mistral, mistral commented out: uncomment the upstream too
MINIMAX_API_KEY minimax, minimax commented out: uncomment the upstream too
ANTHROPIC_API_KEY anthropic, anthropic commented out: uncomment the upstream too

From a source checkout, put the key in the gitignored root .env and restart the stack:

# .env
QWEN_API_KEY=sk-…
make up SERVICES=orchestrator

For a release binary or the desktop daemon, set it in the orchestrator's environment and restart it. A config change alone needs only a reload (POST /admin/v1/reload or SIGHUP), but the environment is read at start.

Check it is enabled and healthy:

curl -s http://127.0.0.1:8787/admin/v1/upstreams -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" \
  | jq '.[] | {name, disabled, healthy}'

Another provider with an OpenAI-compatible API is an upstream of its own; the upstream kinds table says which kind to use, and examples has Azure, LiteLLM and others.

2. An alias that tries the cheapest first

routes:
  - alias: venom-thrifty
    strategy: cheapest
    targets:
      - {upstream: openai, model: gpt-5.4-mini}
      - {upstream: qwen,   model: qwen3.8-flash}
      - {upstream: ollama, model: "${VENOM_OLLAMA_MODEL:-qwen3:8b}"}

cheapest orders the targets by effective price, a 3:1 blend of input and output price, computed when the route table is built: local targets first (they cost $0), then the cloud ones by price, unpriced ones last, ties in the order written. Health checks, circuit breakers and fallback work as on any route, so a down target is skipped and the next cheapest serves. Reload, then look at the order it chose:

curl -s -X POST http://127.0.0.1:8787/admin/v1/reload -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"
curl -s http://127.0.0.1:8787/admin/v1/pricing -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"

GET /admin/v1/pricing lists every model's price and source, the blended price cheapest ranks by, each route's targets in price order, and enabled cloud models nothing prices.

Other strategies that weigh cost:

Strategy Order Tune with
cheapest by price, fixed at load the prices (below)
balanced by a weighted sum of price and live time to first token balance: {price: 0.3, latency: 0.7}
learned the cheapest target whose reported session success is within margin of the best learned: {min_samples: 20, margin: 0.10}; outcomes from POST /admin/v1/sessions/{id}/outcome
fastest by live time to first token only explore

All of them are in routes and strategies.

3. Prices you control

Prices come from the offline price table (prices.yaml next to the config, or VENOM_PRICES), never fetched at runtime. Price a model the table does not know, or one you pay a negotiated rate for, under pricing::

pricing:
  source: [custom, builtin]
  models:
    litellm/my-model:   {input: 0.2, cached_input: 0.05, output: 0.8}   # your own rate, USD per 1M tokens
    azure/my-gpt:       {base_model: gpt-5.4}                            # priced as another model

A target can also carry its own input, cached_input and output, and a route its base_model (pricing).

4. Choose it per key, request or gear

Clients keep asking for the aliases they know; you decide which of them goes cheap.

Per request, call the alias by name: "model": "venom-thrifty".

Per key, without changing the client, re-target its usual aliases with a route profile and make it that key's default:

profiles:
  thrifty:
    description: local first, then the cheapest cloud model
    routes:
      venom-default: [ollama/qwen3:8b, qwen/qwen3.8-flash, openai/gpt-5.4-mini]
      venom-fast:    [ollama/qwen3:4b, qwen/qwen3.8-flash]
curl -s -X PATCH http://127.0.0.1:8787/admin/v1/keys/<id> \
  -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"profile":"thrifty"}'

A profile replaces the alias's targets in the order given and keeps its strategy. A client can also pick it per request with x-venom-profile: thrifty (route profiles).

In the venom CLI, a gear in venom.yaml puts the alias one keystroke away:

gears:
  thrifty:
    key: 10
    description: cheapest target first
    model: venom-thrifty
    thinking: low

Or send the profile from a gear with profile: thrifty (your own gears).

5. Read back what it cost

Every answer names the target that served it:

curl -s http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
  -H "Content-Type: application/json" -D - -o /dev/null \
  -d '{"model":"venom-thrifty","messages":[{"role":"user","content":"ping"}]}' | grep -i '^x-venom'

x-venom-upstream and x-venom-model say who served; x-venom-fallback says why earlier targets did not. Over time:

curl -s "http://127.0.0.1:8787/admin/v1/spend?from=2026-11-01&to=2026-11-30" -H "Authorization: Bearer $VENOM_ADMIN_TOKEN"

returns totals and breakdowns by key, team, model and day, with the costliest sessions; the console's Usage page shows the same. Put a budget on a key so a mistake cannot run up a bill: budget: {daily_usd: 5} on POST /admin/v1/keys or PATCH /admin/v1/keys/{id}. Requests that reach only local models cost $0 and are never refused (keys and budgets).

Troubleshooting

Symptom Cause and fix
The upstream shows disabled Its key variable is empty in the orchestrator's environment. Set it and restart; a reload is not enough
cheapest puts a cloud model last Nothing prices it. Add it under pricing.models; GET /admin/v1/pricing lists unpriced models
403 unpriced_model A key with a hard budget reached a cloud target with no price. Price the model
400 unknown_profile The profile name is not in profiles:; the error lists the configured ones
Requests to venom-default now reach the provider The shipped routes have cloud fallbacks. Remove the cloud target, or use a profile that lists only local targets
The load warns about ttft_timeout A route's first-byte deadline is not shorter than defaults.retry_budget, so no fallback could start. Lower it