Routing, failover drills, local autopilot and shadow tests
Decide which model answers and where it runs, rehearse an outage without sending traffic, move paid work to your own models, and try a candidate model on real requests before you switch.
Four console pages decide and check where your requests go:
| Page | What it is for |
|---|---|
/routing |
Model choice for venom-auto, then the routes: which sources serve each model, in failover order. |
/routing/drill |
The failover drill: what would happen if a provider, key or daemon went down. A dry run. |
/autopilot |
Local autopilot: paid requests your own local models could serve, and a shadow test before you move them. |
/shadow, /shadow/{id} |
Shadow tests: mirror a sample of real requests to a candidate model and compare. |
Routing
/routing is one canvas read left to right. Click any card to change it; the drawer shows its options.
- Model choice (only for requests to
venom-auto): rules in order, joined by "else", then three tiers (Cheap, Balanced, Premium), each a list of models with their list price per 1M tokens and where they are served. The first rule that matches picks the tier; the tier's first model your key may use answers. The Auto for the main role switch also applies it tovenom-defaultunder your main role. - Routes: from Request, a "Model is …" branch for each model with its own route, else the organisation's route. Each route is a list of steps in failover order, then Served or Error.
Test a request takes a prompt (and optionally a role, tools, message count and an image), runs model choice with your unsaved edits, previews the route for the model it picked, and lights the path: rule, tier, model, route step, and the daemon or provider that would serve it. Steps it skipped say why. Nothing is sent to a model.
Below the canvas: what model choice saved in the window, and "What served each model (24 h)": measured p50 latency, error rate and cost per request per model, with list prices marked "catalog" where there is no traffic yet.
Route steps
A route lists one to five steps. A step can be switched off without losing its place; at least one stays on.
| Step | Admits |
|---|---|
private (your daemons) |
Your organisation's own daemons. |
trusted (nodes you trust) |
Public daemons on your allowlist that pass the public mesh checks. Add a node from its card or from the Nodes page. |
public (public nodes) |
Every public daemon that passes the checks, scored by measured quality of service. No cloud provider. |
managed (Venom picks) |
Public daemons as above plus cloud providers. |
cloud (cloud providers) |
Cloud providers only: your provider keys first, then Venom's keys if your organisation allows them. |
A route without managed or cloud never leaves your daemons and the public mesh. cloud alone means no request waits on a daemon; a model with no provider key then fails with 503 no_daemon_available and the message No provider key for "<model>".
| Route option | Values | Default |
|---|---|---|
| Price ceiling | USD per 1M tokens, 0 to 1,000,000 | none. Public nodes and cloud providers priced above it are skipped; your own daemons never are |
| Preference | balanced, cheapest, fastest, local, cloud |
balanced. It changes the order inside a step, never the step order. fastest orders provider keys by the first-token time your own traffic measured |
A change reaches every cloud instance within 10 seconds. With no route set, requests go to your daemons, then the cloud. The Daemons page has a one-click version for the organisation route: daemons then cloud, daemons only, or cloud only.
A request can narrow its route, never widen it, with x-venom-profile (cloud, cloud-first, daemons, daemons-first); a gear role's route sends it. A profile the route doesn't allow fails with 403 profile_forbidden.
curl https://cloud.usevenom.dev/v1/chat/completions \
-H "Authorization: Bearer vck_…" \
-H "x-venom-profile: daemons-first" \
-H "Content-Type: application/json" \
-d '{"model": "venom-auto", "messages": [{"role": "user", "content": "Summarise this diff"}]}'The answer's x-venom-auto-tier and x-venom-auto-reason headers say which tier and rule model choice used.
Failover drill
/routing/drill answers "what if this went down?" without anything going down.
- Under What goes down, pick a provider, one of your keys, a daemon, one model at a provider, or a whole group.
- Press Run drill.
For each model your organisation used last week, each gear role and each budget, the page shows whether it fails over and to what, or stops answering, with the price and first-token change and red flags. The totals:
| Figure | Meaning |
|---|---|
| Models affected | Models the outage touches, of those walked. |
| Stop answering | Models with no fallback left, and their requests per day. |
| Requests/day moved | At last week's pace. |
| Extra cost | What the pricier fallbacks would add per day while it lasts. |
| Red flags | Warnings and critical findings. |
The drill itself sends nothing. The optional live probe sends one tiny request to each fallback to prove it answers: at most 5 requests, 8 output tokens each and $0.05 in all, after a confirm.
Local autopilot
/autopilot finds paid requests your own local models could serve. It reads the role and gear the venom CLI names on each request (classifier, summary, reflex and worker requests), so it fills in once CLI traffic comes through the cloud.
- Totals: what moving could save a week, the part proven by shadow tests (the local model agrees 90% or more), paid role requests per week, and the local models served now.
- Suggestions: per gear role, last week's requests and cost, the local model, how well it fits, and what a shadow sample says. Apply sets that gear role to the local model, daemons first; Revert under Applied puts it back.
- Shadow evaluation: off until an admin turns it on. It replays captured requests on your local model every 5 minutes, through your daemons, and compares the answers.
- Latest samples.
| Shadow setting | Rule |
|---|---|
| Shadow evaluation | on or off; off by default |
| Replays a week | at most 1,000 |
| GPU $ an hour | 0 leaves the hardware out of the savings |
| Judge model | optional, for free-text answers; counts toward the quality checks' monthly cap |
Shadow evaluation needs stored request content (the key vault and content logging on) and a local model. The page says what is missing.
Shadow tests
A shadow test mirrors a share of matching real requests to a candidate model after each client already has its answer, so the client never waits for it. It compares agreement, latency and cost, and recommends a switch when quality is equal and the candidate is neither dearer nor slower.
- Turn Shadow routing on under Settings on
/shadow(off by default). - New shadow test: match by model, gear or role (at least one), name the candidate model, set the caps, Start.
- Open the test at
/shadow/{id}as samples come in. A verdict needs 30 graded samples. - Switch sets the matched gear roles to the candidate and stops the test; Revert puts them back.
| Field | Rule | Default |
|---|---|---|
| Model, Gear, Role | what to match; at least one | any |
| Candidate model | the model to try; not the one already used. Its calls go through your routing, keys and daemons | required |
| Sample % | share of matching requests, above 0 and at most 100 | 5 |
| $ a day | shadow calls and judge, above $0 and at most $1,000 | 2 |
| At once | shadow calls in flight, 1 to 8 | 2 |
| Name | at most 80 characters | made from the match |
| Judge model | organisation setting; grades free-text answers against the primary's | off: deterministic checks only |
An organisation may have at most 20 tests. On /shadow/{id}, Edit caps changes the name, sample, daily cap and concurrency (the match and the candidate are fixed); Stop and Resume; Delete removes the test and its samples. Shadow calls are booked as source = 'shadow': they count in spend and show in Requests, but not in the figures that skip internal traffic.
Who can do what
| Action | Permission | Roles |
|---|---|---|
| See all four pages, test a request, run a drill | routing:read |
owner, admin, developer, viewer |
| Change routes, steps, the allowlist | routing:write |
owner, admin, developer |
| Change model choice; live probe; autopilot Apply, Revert and settings; shadow tests and settings | org:manage |
owner, admin |
Troubleshooting
| Message | Cause and fix |
|---|---|
| Pick at least one mode. | A route needs one step switched on. |
| Your role cannot change routing. | Your role lacks routing:write. |
403 profile_forbidden from /v1 |
The request's x-venom-profile asks for a source its route doesn't allow. Drop the header or change the route. |
503 no_daemon_available, No provider key for "<model>" |
The route is cloud only and no key serves that model. Connect one on the Providers page. |
| Nothing to drill yet | No traffic, gears or routes that a provider or daemon serves. |
| Autopilot: Nothing to move yet | No paid role requests from the venom CLI last week. |
| Match at least one of a model, a gear or a role | A shadow test never mirrors all traffic. |
| Shadow routing is off: tests are paused | Turn it on under Settings. |
Related
- Organisation gears: the gear roles that autopilot and shadow tests switch.
- Providers and provider keys: the cloud sources a route can use.
- Budgets: spend limits that can block or downgrade a request.
- Gears and route profiles:
profile:in the CLI.