---
title: Routes and strategies
description: Define model aliases, choose how their targets are ordered, and understand fallback, retries, stickiness, experiments and shadow traffic.
section: orchestrator
order: 20
audience: [operator, builder, agent]
sources: [services/go-orchestrator/internal/config/config.go, services/go-orchestrator/internal/config/shadow.go, docs/architecture.md, services/go-orchestrator/AGENTS.md, config/orchestrator.yaml]
features: [orchestrator.config-routes]
---

A route gives a model alias (the `model` a client sends) an ordered list of targets, each a model on a configured upstream. The alias is what clients depend on; the targets are yours to change.

## Minimal example {#example}

```yaml
routes:
  - alias: venom-fast
    strategy: fallback          # the default
    ttft_timeout: 12s           # give up on a target that hasn't sent a byte by then
    targets:
      - {upstream: ollama, model: "qwen3:4b"}
      - {upstream: openai, model: gpt-5.4-nano}
      - {upstream: qwen,   model: qwen3.8-flash}   # skipped while QWEN_API_KEY is unset
```

```bash
curl -s http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
  -H "Content-Type: application/json" -D - -o /dev/null \
  -d '{"model":"venom-fast","messages":[{"role":"user","content":"ping"}]}' | grep -i x-venom
```

The `x-venom-upstream` and `x-venom-model` headers name the target that served; `x-venom-fallback` lists why earlier ones did not.

## Route keys {#keys}

| Key | Type | Default | What it does |
|---|---|---|---|
| `alias` | string | required | The name clients use. Unique. |
| `strategy` | see below | `fallback` | How targets are ordered for each request. |
| `targets` | list | required (not with `experiment`) | `{upstream, model, weight}` entries; `weight` (default 1) is used by `weighted`. A target may also carry `base_model`, `input`, `cached_input`, `output` to price it. |
| `ttft_timeout` | duration | the upstream's `timeout` | Per-attempt deadline for the first byte. It only applies while another target remains, and only tightens the upstream's own timeout. Keep it well under `defaults.retry_budget` (the load warns otherwise). |
| `images` | alias | none | Another ordinary route that serves this alias's requests carrying an image, for an alias whose own models are text-only (the shipped `glm` sends images to `glm-vision`). |
| `rate_limit` | `{rpm, tpm}` | none | A token bucket for all keys together on this alias: requests and tokens per minute (0 = no limit on that axis). |
| `base_model` | model id | none | The model every target is priced as unless it declares its own price. |
| `learned` | `{min_samples, margin}` | `20`, `0.10` | Tunes `strategy: learned` (an error on other strategies). |
| `balance` | `{price, latency}` | half and half | Weights for `strategy: balanced` (an error on other strategies; both 0 is an error). |
| `explore` | 0–0.5 | `0.05` | Share of a `fastest` or `balanced` route's requests sent first to the target measured longest ago, so a recovered target is seen again. |
| `experiment` | `{variants}` | none | Makes the alias an A/B test (below). Excludes `targets`, `strategy`, `ttft_timeout` and `images`. |
| `shadow` | map | none | Mirrors a sample of the alias's chat completions to a candidate model (below). |

## Strategies {#strategies}

| Strategy | Order |
|---|---|
| `fallback` | As configured. The first servable target serves. |
| `round_robin` | Rotates the starting target per request. |
| `least_latency` | By each upstream's moving-average latency. |
| `weighted` | Random, in proportion to each target's `weight`. |
| `cheapest` | By effective price (a 3:1 input:output blend from `pricing:`, `base_model` and the price table), computed when the table is built: local first, unpriced last, ties in configured order. |
| `fastest` | By live time to first token (p50 and p90 per upstream/model, from streamed traffic; unmeasured targets first), with `explore` traffic to the longest-unmeasured target. |
| `balanced` | By a weighted sum of price and live time to first token, each scaled across the route's targets (`balance`). |
| `learned` | By reported session outcomes against cost over the last 30 days (`POST /admin/v1/sessions/{id}/outcome`); the configured order while a target has fewer than `min_samples` sessions. A target qualifies within `margin` of the best success lower bound, and the cheapest qualified one goes first. |

`GET /admin/v1/routes` shows every alias, its targets in order, each target's live time-to-first-token estimate and whether it can serve now; `GET /admin/v1/pricing` shows the prices `cheapest` ranks by.

## Fallback and retries {#fallback}

- **Before the first byte only.** A connection error, `429`, `5xx` or a missed TTFT deadline moves to the next target with full-jitter backoff. Once a byte reaches the client, a failure ends the stream with an error event and the client decides.
- **Known-down targets cost nothing.** A target whose upstream failed its last health check, whose circuit breaker is open, whose upstream lacks the model, or which sits out a quota cooldown is skipped without an attempt while another target can serve. The last remaining target is still tried, so a lone local model off the grid answers.
- **Slow is not down.** A missed TTFT deadline does not count against the breaker while the health check passes. A target that served this session in the last 5 minutes gets four times the deadline (never past `timeout`).
- **Retries don't multiply.** Every target gets one attempt; same-upstream retries come after, within `defaults.max_attempts`, `defaults.retry_budget` and the per-upstream retry budget (`retry_ratio`, `retry_min_per_second`). A request from an OpenAI SDK that is itself retrying (`x-stainless-retry-count` > 0) gets no same-upstream retry, and a `Retry-After` longer than `defaults.backoff_max` is returned to the client rather than slept on (unless rate shaping can queue it).
- **Context windows are checked, never truncated.** A target whose window can't hold the request (prompt bytes/4 plus max output) is passed over for a larger one. If none can, the answer is `400 context_length_exceeded` before any attempt.

`x-venom-fallback` reason codes (stable): `429`, `429_quota` (quota or balance exhausted; the target cools down), `quota_cooldown`, `circuit_open`, `unhealthy`, `busy` (at `max_inflight`), `unsupported` (the target can't serve this request), `context_window`, `missing` (the upstream lacks the model), `timeout`, `5xx`, `404`, `auth`, `conn_error`, `context_length`, `rejected_param` (the upstream refused a field or value; the 400 is held for the client if no later target serves), `rate_shaped`. When a target was passed over for its context window and both are priced, `x-venom-context-cost-usd` gives the input-cost difference.

## Session stickiness {#sessions}

Send `x-venom-session: <id>` (the venom CLI does on every request) and the orchestrator keeps the session on one target by rendezvous hashing over the healthy targets, so engine prefix caches and provider prompt caches are reused. Subagents send `x-venom-parent-session` too, so spend rolls up under the parent. Without the header, the session is a hash of the body's `prompt_cache_key`, else `safety_identifier`, else `user`. On llama.cpp builds without a host prompt cache a session gets a slot of its own; on OpenAI a `prompt_cache_key` is chosen from a pool sized to the model's traffic (a key the client set is kept). Side requests (`x-venom-session: <session>:<kind>`) get the body defaults in `side_requests:` ([Profiles and gears](/docs/orchestrator/profiles-and-gears.md#side-requests)).

## A/B experiments {#experiments}

```yaml
- alias: venom-ab
  experiment:
    variants:
      - {name: default, target: venom-default, weight: 50}
      - {name: smart,   target: venom-smart,   weight: 50}
```

| Variant key | Default | What it does |
|---|---|---|
| `target` | required | Another ordinary route's alias (routed with its own strategy and deadlines) or `upstream/model`. Not another experiment. |
| `name` | the target | 1–128 of `A-Z a-z 0-9 . _ : / @ + -`; sent in `x-venom-variant`. |
| `weight` | 1 | Share of new sessions; 0 stops new sessions (existing ones keep their variant). |

Assignment is sticky per session (subagents inherit their parent's), by weighted rendezvous hashing, so every orchestrator agrees and adding a variant only moves the share it takes. Every answer carries `x-venom-variant`. Assignment is not a security boundary: a client can pick a variant by trying session ids, so only put variants in one experiment that every key using the alias may reach. Results: `GET /admin/v1/experiments` and `GET /admin/v1/experiments/{alias}?since=<RFC 3339>` ([Admin API](/docs/orchestrator/admin-api.md#experiments)).

## Shadow traffic {#shadow}

```yaml
- alias: venom-default
  targets: [...]
  shadow:
    candidate: qwen/qwen3.8-max   # an alias, upstream/model or bare id
    sample: 0.05                  # share of the alias's chat completions, (0, 1]
    daily_usd: 2                  # required: no replay starts once the UTC day's shadow cost reaches it
```

After the client has its answer, a sampled chat completion is replayed on the candidate and the two answers are compared; the client only ever sees the primary. `max_concurrency` (default 1, at most 8) replays run at once, `queue` (default 16, at most 256) samples may wait, and a primary answer longer than `max_capture_bytes` (default 1 MiB) drops the sample. Replays carry no client key, are logged with source `shadow`, and are summarised by `GET /admin/v1/shadow?days=7&route=<alias>`.

## Errors {#errors}

| Status | Code | Cause |
|---|---|---|
| 400 | `image_input_unavailable` | the request carries an image and no target (nor the `images` route) takes images |
| 400 | `context_length_exceeded` | no target's window holds the request |
| 404 | `model_not_found` | unknown alias, upstream or model |
| 429 | `rate_limit_exceeded` | the route's `rate_limit` (or the key's) |
| 503 | `model_unavailable` | every remaining target lacks its model; each is named with its fix |
| 503 | `no_upstream` | no target is enabled; names the missing credential |

## Related

- [Upstreams and kinds](/docs/orchestrator/upstreams.md), [Profiles and gears](/docs/orchestrator/profiles-and-gears.md), [Configuration reference](/docs/orchestrator/configuration.md#defaults) for the retry and breaker settings.
