Skip to content
Venom router · in the daemon and in Venom Cloud

Every request goeswhere its cache is warm.

An agent re-sends a long, growing prompt on every turn. The router keeps each session on the GPU that already holds that prompt, moves it only when the move pays for itself, and fails over before the first byte. It does this in microseconds, and every claim on this page comes with how we measured it.

  • +49 µs1

    added per request at p50, auth and routing included

  • 100%2

    of a session's requests on its warm daemon at 10% load, up from 61-64%

  • 7 µs3

    for the cloud to pick among 10 daemons, p50

  • 1,0004

    concurrent streams with 0 errors and nothing buffered

Two places, one idea

One router on your machine, one across your machines

Both pick a target per request with the same goal: the fewest milliseconds to the first token, without piling every request onto one backend.

The orchestrator

in the daemon

A Go gateway in front of your vLLM, SGLang, llama.cpp and Ollama servers and your provider keys. It resolves a model alias to an ordered list of targets and adds 49 µs at p50.

  • Fallback to the next target before the first byte, circuit breakers per target
  • Session pins and a radix tree of served prompts for prefix-cache hits
  • Retry budgets, so its retries don't stack on the SDK's
  • Priority classes for busy GPU slots, interactive work weighted 8:3:1 over background jobs
  • Streams relayed as written, never buffered

Venom Cloud

across daemons

A pure function over the daemons that are online for your team, their leases, load and measured TTFT. It picks one in 7 µs at p50 for 10 daemons, then the request goes direct or through the relay.

  • Rendezvous hashing with bounded loads keeps a session on its warm daemon
  • Power of two choices, so stale load reports don't send everyone to one daemon
  • A score in milliseconds of TTFT, with a bonus worth the re-prefill a move would cost
  • Hedged short requests, under 5% of a key group's traffic
  • Hybrid logical clocks, so a request's trace stays in order

Under the hood

The algorithms, and what each one bought

Each card names the technique, says what it does for your requests, and gives the result we measured. Where a number comes from a simulation, the card says so.

  1. Rendezvous hashing with bounded loads

    A session stays where its prompt cache is warm2

    Each session ranks the daemons by a 64-bit rendezvous hash. It goes to the first one under its load cap, the bound from Mirrokni, Thorup and Zadimoghaddam with a floor for idle slots. A daemon that leaves moves only its own sessions, and a full one spills along each session's own order.

    61-64% → 100%

    of requests on the session's warm daemon at 10% load; 64-68% → 81-86% at 50%, with no added queueing

    Simulated: 10 daemons × 4 slots, Zipf-popular sessions, 50k requests.

  2. Power of two choices

    No stampede onto the daemon that looked idle5

    Load reports can be 10 s old, and picking the least loaded daemon from a stale snapshot sends every request to the same one. The router samples two daemons and keeps the session's owner only if it costs no more than both.

    400 → 21

    requests on the busiest of 20 daemons in a 400-request burst (a fair share is 20)

    Simulated burst between two heartbeats.

  3. Cost in milliseconds of TTFT

    Moving a long session has to pay for its re-prefill6

    Every candidate gets a score in expected milliseconds to the first token: network, queue, error rate and breaker state. The owner's bonus is the cold prefill a move would cost, from each daemon's own running average of prefill time per token, so a 50k-token session doesn't leave a warm GPU to save a second of queueing.

  4. Radix tree of served prompts

    A new session goes to the backend that already holds its prefix7

    On a route with several self-hosted backends, the orchestrator keeps a bounded radix tree of the prompts each one served, in 512-byte blocks. A new session lands on the backend with the longest match, unless that backend is busier than the rest.

    75.5% → 79.7%

    prefix-cache hit rate, for +0.62 µs and one allocation per request

    Simulated on 3 fake vLLM targets with tight caches.

  5. Hedged requests

    Short requests don't wait on a slow daemon8

    A short non-streamed request, an embedding or a systemone call goes to a second daemon once the first passes its own p95 time to first token. The first answer wins, the other is cancelled, and you are billed once. Hedges stay under 5% of a key group's traffic.

  6. Retry budgets and backpressure

    Retries that can't turn an outage into a storm9,10

    Same-upstream retries draw on a budget of 20% of first attempts, and a retry the OpenAI SDK is already making isn't retried again underneath it. A connection reset counts as backpressure, not a dead upstream, so a cold start no longer trips the breaker.

    30 → 13

    upstream calls for 10 requests to a failing upstream; breaker trips on a cold start went from 4 of 14 to 0

    Orchestrator tests and load tests on a shared M2 Pro.

  7. Streaming without buffering

    The first token reaches you when the model writes it11,12

    Bytes go to the client as the upstream writes them, and every stream tells proxies such as nginx not to hold events back. Request bodies are rewritten by scanning and splicing the JSON in place, not by decoding and re-encoding it.

    5.05 s → 0.01 s

    to the first event behind nginx with gzip; a 525 KB body rewritten in 285 µs instead of 1.60 ms

    make smoke against the prod image, and make bench on an M2 Pro.

  8. Weighted priority classes

    Interactive requests get most of a busy backend's slots13

    When a backend is at its concurrency limit, waiting requests are served by class: interactive, then merge checks, then background work, by smooth weighted round robin at 8:3:1, first come first served within a class.

  9. Hybrid logical clocks

    A request's trace is in the right order even when clocks drift14

    Every step of a job, on the cloud and on each daemon, carries a hybrid logical clock stamp, so a failover or a resumed session reads in causal order on the request's timeline.

    6,202 → 0

    of 46,373 causal pairs out of order, wall clock against the stamps

    Simulated: 600 jobs, daemon clocks off by up to ±5 s.

  10. Space-Saving and HyperLogLog++

    Usage questions answered in milliseconds15

    Top spenders, top models and distinct sessions come from mergeable sketches kept in hourly and daily rollups, not from scanning the request log. Money totals stay exact.

    1,629 ms → 155 ms

    at 1M rows, distinct sessions within 0.28% and every top-10 entry found

    Postgres, 31 days of rows, scripts/bench-usage-sketches.ts.

  11. CUSUM over audits

    Public nodes that swap in a cheaper model get flagged16

    Venom replays a sample of a public node's requests on a reference and scores how far the answers drift. A cumulative sum over those scores alarms on small, steady drift that no single audit would fail, and suspicion raises the audit rate.

    13 audits

    median to flag Q6_K served in place of Q8_0; the honest node never alarmed in 3,000

    Qwen3-4B on llama.cpp, an RTX 5070 Ti, bootstrapped audit sequences.

Where a request may go

You decide which tiers a route may use

Your apps call one URL. The cloud checks the key, its limits and budget, then sends the request down the tiers your routing mode permits, in this order.

  • venom CLI
  • pyOpenAI SDK
  • Any HTTP client
Venom Cloud

cloud.usevenom.dev/v1

  • key auth
  • RPM / TPM
  • budget
  • routing mode
  • trace
  1. 1
    Your daemonsprivate

    GPU boxes running the orchestrator in mesh mode. They dial out to the cloud, so no inbound ports, and their provider keys never leave them.

  2. 2
    Trusted public nodestrusted

    Hosts from the public catalogue that you allowlist by name, ranked by trust and by QoS the cloud measures itself.

  3. 3
    Your provider keysBYOK

    OpenAI, Z.ai, Qwen (DashScope) and TypeSafe keys sealed in your org's vault. Several per provider, tried by priority, with failover on 401, 429 and 5xx.

  4. 4
    Venom's keysmetered

    When your plan allows it, the cloud can serve on its own provider keys and bill the usage. Nothing to set up.

Your routing mode decides which tiers a route may use. If a target fails before the first byte, the request moves to the next one.

Private

private

Only your own daemons serve the route. A model that would reach a cloud provider is left out, so the request stays on your hardware or fails.

Trusted nodes

trusted

Browse the public node catalogue and allowlist the hosts you want, by name. Nothing outside that list serves the route.

Managed

managed

One endpoint, and Venom picks the node, public daemon or provider for each request by price, latency and trust.

Set a mode for the whole org or per route alias, and combine up to three in the order you want them tried.

Measured

The numbers, with their conditions

From make bench, load tests against a fake vLLM upstream, live runs and the cloud's router benchmarks. They show what the router costs and what each change bought. They are not head-to-head runs against other gateways.

+49 µs1

added per request, p50

Auth, alias resolution, quirks, usage accounting and session affinity on the path. Streaming adds +65 µs p50, and nothing is buffered.

make bench on an Apple M2 Pro with a local fake upstream.

1,0004

concurrent streams, 0 errors

+0.4 ms TTFT at p50, throughput 0.997× of the upstream alone, 163 MB peak memory.

Fake vLLM upstream on a shared M2 Pro.

0% → 95%17

cached on a new session's first request

Sessions share a small pool of OpenAI cache keys, so a new session finds the common prompt prefix already cached.

Live, eight new sessions with the same 5k-token prefix.

0% → 88-95%18

prompt reused on llama.cpp

The orchestrator leaves slot choice to builds with the host-RAM prompt cache instead of pinning, so interleaved sessions keep their cache.

llama.cpp b11459, measured through the orchestrator.

75.5% → 79.7%7

prefix-cache hit rate, simulated

A new session goes to the backend that holds the longest prefix of its prompt, unless that backend is busy.

Simulation on 3 fake vLLM targets with tight caches.

7 µs3

for Venom Cloud to pick a daemon, p50

11 µs at p95 across 10 eligible daemons, 65 µs across 100. Routing is not where a request's time goes.

Wall clock on Node 24.

The full comparison, with every speedup

Sources

  1. 1.make bench, Apple M2 Pro, a local fake upstream, a ~5 KB chat request with a tool; the added time includes the extra loopback hop. Re-runs on a loaded machine measured +59 to +63 µs. services/go-orchestrator/AGENTS.md "chat body with a tool; non-stream: one JSON response;".
  2. 2.Bounded-load session owner, simulated (npm run sim:bounded): 10 daemons × 4 slots, Poisson arrivals, 1,000 Zipf-popular sessions (s = 0.8 and 1.1), 50k requests. A simulation, not a production measurement. docs/cloud-mesh.md "(`bun run sim:bounded`, `sim/bounded-load.ts`). The setup".
  3. 3.Venom Cloud's route() wall-clock time on Node 24 for 10 eligible daemons (65 µs p50 for 100). docs/cloud-mesh.md "Only the `route()` timings are wall-clock measurements", docs/cloud-mesh.md "`route()` itself takes".
  4. 4.Load test against a fake vLLM upstream (300 ms TTFT, 100 tokens/s) on a shared M2 Pro; numbers are at 1,000 concurrent streams. services/go-orchestrator/AGENTS.md "tokens/s per stream,", services/go-orchestrator/AGENTS.md "**Streaming levels**".
  5. 5.Power-of-two-choices pick, simulated: a burst of 400 sessionless requests between two heartbeats on 20 daemons (asserted in sim/chaos.test.ts). docs/cloud-mesh.md "sessionless requests between two heartbeats put all".
  6. 6.Venom Cloud's score: expected TTFT in ms (network, queue, error penalties) less an affinity bonus worth the prefill a move would cost, up to 20 s, from each daemon's own EWMA of prefill time per uncached token. docs/cloud-mesh.md:795-831.
  7. 7.Longest-prefix routing for new sessions, simulated on 3 fake vLLM targets with tight caches and 8 prompt families. A simulation, not a production measurement. CHANGELOG.md "unless that target is busy against the others (SGLang's balance".
  8. 8.Venom Cloud's hedging: short non-streamed requests, embeddings and systemone only; sent to the runner-up after the chosen daemon's p95 TTFT; under 5% of a key group's requests a minute; the loser is cancelled and the request billed once. Tested in test/db/hedge.test.ts. docs/cloud-mesh.md:856-895.
  9. 9.Retry budget test: 10 requests to an upstream that always answers with a 503 error, max_attempts 3 (server.TestRetryBudgetCapsRetries). services/go-orchestrator/AGENTS.md "`server.TestRetryBudgetCapsRetries` (".
  10. 10.Load test: a fresh orchestrator hit by 1,000 streams at once, 14 interleaved trials, before and after connection resets were treated as backpressure. services/go-orchestrator/AGENTS.md "**Failure behaviour under load**".
  11. 11.make smoke STAGES=prod: nginx 1.30.5 with gzip in front of the prod image, a ~5 s stream; time to the first event with and without the orchestrator's X-Accel-Buffering header. services/go-orchestrator/AGENTS.md "`make smoke STAGES=prod` checks it end to end".
  12. 12.Request rewrite on a ~525 KB body (a long agent session): a JSON scan and splice against a decode and re-encode, make bench, Apple M2 Pro. services/go-orchestrator/AGENTS.md "Request rewriting on a".
  13. 13.The orchestrator's priority classes: interactive, blocking-merge and background requests share an upstream's slots by smooth weighted round robin, 8:3:1. services/go-orchestrator/AGENTS.md:1668-1671.
  14. 14.Hybrid logical clocks, simulated (sim/hlc-skew.ts): 600 jobs, 30% failovers, 20% session resumes, three function instances, daemon clocks off by up to ±5 s. docs/cloud-mesh.md "restore before the old daemon's last event".
  15. 15.Usage sketches (Space-Saving, HyperLogLog++) against exact GROUP BYs, 1M request rows over 31 days in Postgres (scripts/bench-usage-sketches.ts). services/nodejs-cloud/AGENTS.md "- **Measured** (`node scripts/bench-usage-sketches.ts`".
  16. 16.Public-mesh audits of Qwen3-4B on llama.cpp, Q8_0 registered, on an RTX 5070 Ti; CUSUM with k = 4, h = 10,000, median over 400 bootstrapped sequences of the measured audits. docs/public-mesh-hardening.md "### Real quantisations (llama.cpp, Qwen3-4B)".
  17. 17.OpenAI prompt_cache_key pooled across sessions, measured live: eight new sessions with the same 5k-token prefix. CHANGELOG.md "live, eight new sessions with the same".
  18. 18.llama.cpp b11459 with the host-RAM prompt cache, interleaved sessions, measured through the orchestrator. CHANGELOG.md "pinning each session to a slot with `id_slot`".

Put the router in front of your GPUs

The daemon runs it on your machine for free. Venom Cloud runs it across every daemon your team connects.