How the private mesh works
How Venom Cloud routes a key group's requests to its own daemons, directly with a 307 and a ticket or relayed through the cloud, and what each side sees, signs and keeps.
Venom Cloud is a control plane: stateless functions plus one Postgres database. It authenticates the client, applies rate limits and budgets, picks a daemon, and either sends the client straight to that daemon or carries the stream itself. The daemon, venom's Go orchestrator, is the data plane. The normative contract is docs/cloud-mesh.md in the source tree, and the decision record is ADR 0017; this page describes what an operator or a client author needs from them.
The parties
| Party | What it is | Talks to |
|---|---|---|
| Client | the venom CLI or any OpenAI SDK, with a vck_… client key |
the cloud's /v1; on the direct path, also a daemon's /v1 with a ticket |
| Cloud | Next.js functions (on Vercel or next start) and Postgres |
clients; daemons only through the requests daemons make to it |
| Daemon | the orchestrator in mesh mode, with its own engines and keys | the cloud's daemon API, outbound only (/api/mesh/v1/*, see the daemon API reference) |
A daemon belongs to the key group whose join token enrolled it. A key group holds API keys, daemons, policies and GPU grants, not people; the API and the daemon protocol call it a team (team_id, accept.teams, VENOM_MESH_ACCEPT_TEAMS). Other key groups of the same tenant can use a daemon through a share (console: Daemons, the daemon, Shares; optionally only some models, with a weight) or a GPU grant (all of a key group's daemons, with a daily token quota and an end date). There is no sharing across tenants; the public mesh is the separate, verified pool for that.
Identities and tokens
| Thing | Format | Lifetime |
|---|---|---|
| Client key | vck_… |
until revoked |
| Join token | vjt_…, a number of uses |
default 1 hour, at most 7 days; 1 use from the console |
| Daemon key pair | Ed25519; the private key is $VENOM_DATA_DIR/mesh/daemon.key (mode 0600) and never leaves the machine |
until rotated (orchestrator mesh rotate) |
| Daemon request JWT | compact JWS, alg: EdDSA, kid = daemon id, bound to the method, path and body hash |
at most 60 s, with ±30 s of clock skew |
| Cloud signing keys | Ed25519, in the cloud's VENOM_MESH_SIGNING_KEYS; published at GET /api/mesh/v1/jwks |
rotated by adding a key first |
| Ticket | vtk. + a JWS the cloud signs |
60 s and single use per request; 10 min for a session |
The join token is spent once, at enrollment: the daemon sends it with its new public key and gets back a daemon id. From then on every daemon call is signed with the private key. The cloud stores only the public key, so nothing it holds lets someone act as the daemon.
Two paths for a request
Every client request first reaches the cloud's /v1. After authentication, limits and a budget pre-charge, the router picks a daemon and a path:
| Path | How it works | Added time to first token |
|---|---|---|
| Direct | The cloud answers 307 Location: <daemon endpoint><path>?venom_ticket=vtk.…. The client re-sends the same body to the daemon, which checks the ticket offline and streams the answer straight back. The daemon reports usage afterwards with POST /api/mesh/v1/usage. |
one client-to-cloud round trip plus about 5 to 20 ms of routing; nothing per token |
| Direct, mesh-aware | The client calls POST /v1/mesh/route once per session and reuses the session ticket (Authorization: Bearer vtk.…), refreshing at 80% of its life. |
nothing after the first request |
| Relay | The cloud queues a job, the daemon's open channel receives a dispatch, the daemon acks and posts its response in event batches, and the cloud streams them to the client. |
roughly RTT(cloud to daemon) + RTT(client to cloud) − RTT(client to daemon) + 20 to 40 ms; per token at most a 20 ms batch window |
The OpenAI Python SDK and most others follow a 307 and re-send the body, so the direct path works with plain SDKs. A request over 4 MiB cannot pass through the cloud (413 request_too_large) and needs the direct path.
Which path is used
A daemon can take direct requests only when it advertises an endpoint (VENOM_MESH_ENDPOINTS) the client can reach:
| Endpoint kind | URL rule | Reachable by |
|---|---|---|
public |
https:// |
every client |
tunnel |
https:// (for example Cloudflare Tunnel or Tailscale Funnel) |
every client |
tailnet |
http:// allowed on a 100.64.0.0/10 address or a *.ts.net name; network required |
clients that send x-venom-networks: tailnet:<name> |
lan |
http:// allowed; network required |
clients that send x-venom-networks: lan:<label> |
The client has to send x-venom-networks itself. The contract names a CLI variable for it, VENOM_NETWORKS=tailnet:acme.ts.net, but the current venom CLI does not read it, so from the CLI only public and tunnel endpoints are reached directly. A daemon with no reachable endpoint is relayed.
In the default auto mode the router also compares the two paths per request: it estimates the direct round trip from the client's and the daemon's locations and the relay's cost from the cloud's measured dispatch-to-ack time, and relays when that is faster. A client far from a daemon that sits next to the cloud often gets its first byte sooner through the relay. When a location is unknown the path stays direct. The daemon's location is VENOM_MESH_GEO, or a region inferred from its time zone; it never asks a geo-IP service.
| Request header | Values | Effect |
|---|---|---|
x-venom-mesh-mode |
auto (default), direct, relay |
direct fails with 503 no_direct_path when no endpoint is reachable; explicit values are never second-guessed |
x-venom-networks |
tailnet:<name>,lan:<label> |
networks the client can reach |
x-venom-daemon |
a daemon name or id | pins the request to that daemon, if the key may use it; a down daemon then gives 503 |
x-venom-session, x-venom-parent-session |
a session id | keeps a session on the same daemon so its prefix cache stays warm |
Response headers say what happened: x-venom-mesh-mode (direct or relay), x-venom-daemon, x-venom-daemon-region, x-venom-job, x-venom-route-reason (affinity, p2c, argmin, pinned, session_pin, cloud, hedge, failover:<reason>) and x-venom-served-by (daemon or cloud). The daemon's own x-venom-upstream and x-venom-model are passed through. An org whose prompt filters could act on a session is always relayed, so the filters see every prompt.
Picking a daemon
The router keeps only daemons that are live (heartbeat within the lease), accept the key's key group, serve the model with the needed features (vision, tools, context window), and are not held off by their circuit breaker. With a session id it picks by rendezvous hashing with bounded load, so a session stays on its daemon until that daemon is over its fair share. Without one it compares expected time to first token (network round trip, queue, recent errors) over two random candidates. When no daemon is eligible and the org has cloud provider keys configured, the cloud's own providers serve the request (x-venom-served-by: cloud).
Before the first byte reaches the client, a relayed job moves to the next candidate when the daemon does not ack within 2 s, fails with a 429 or 5xx, or loses its job lease: at most 3 attempts within 30 s. After the first byte nothing is retried; the client gets a terminal SSE error event (daemon_lost, daemon_error, stream_time_limit) and can retry or resume with GET /v1/mesh/jobs/{job_id}/stream?after=<seq> for 120 s after the end.
What stays on the daemon
- Provider keys, engine URLs and the orchestrator's own client keys. The cloud's dispatch carries the client's JSON body and an allowlist of headers, never
Authorization. - The private key and the per-key-group cache salt (
$VENOM_DATA_DIR/mesh/cache-salt.key). Requests from key groups other than the daemon's own get acache_saltderived from it onvllmandsglangupstreams, so one key group cannot time another's prefix cache. - Content on the direct path. The daemon sends usage only; it sends request and response bodies (
POST /api/mesh/v1/content) only when the ticket says the key's org logs content. - The local
/v1. Its keys, budgets and request log work as before; mesh requests appear in the log with the key namemesh:<name>.
The daemon checks each ticket against the cloud's published keys (cached for an hour, refreshed on an unknown kid), that it is the ticket's audience, that the key group is in its accept set (and in VENOM_MESH_ACCEPT_TEAMS when set) and that the model is allowed. A request ticket's id is remembered until it expires, so it cannot be replayed.
Liveness
| Constant | Value |
|---|---|
| Heartbeat | every 10 s (stretched for large fleets: live daemons ÷ 50 per second) |
| Suspect | no heartbeat for 20 s; deprioritised, not dropped |
| Lease | 30 s; no new work after it lapses |
| Channel | an SSE stream the cloud closes at its function's limit minus 30 s (270 s on Vercel Hobby, 770 s on Pro); the daemon opens the next one 15 s early |
| Long-poll fallback | GET /api/mesh/v1/poll, 25 s |
| Ack timeout | 2 s from dispatch |
| Retired | 14 days offline: hidden, still revocable |
The timing constants live in services/nodejs-cloud/contracts/timing.json, which the cloud and the Go daemon both test against.
enrolled ──first heartbeat──▶ active ⇄ suspect ──no heartbeat 30 s──▶ offline ──14 d──▶ retired
active ──drain (console, SIGTERM)──▶ draining ──jobs done──▶ offline
any ──revoke──▶ revoked (terminal; the daemon gets 401 daemon_revoked and stops)When the cloud is unreachable, the local /v1 keeps serving, direct tickets already issued keep working until they expire, and usage reports queue in memory (up to 10,000 or one hour). On a restart, jobs that had sent no byte are re-dispatched by the cloud.
Errors clients see
| Status | code |
Meaning |
|---|---|---|
| 404 | model_not_found |
no daemon serves the model, and none ever advertised it |
| 413 | request_too_large |
over 4 MiB through the cloud; use the direct path |
| 502 | daemon_error |
every attempt failed before the first byte |
| 503 | no_daemon_available |
none eligible; reasons lists why (offline, breaker, model, team for the key group) |
| 503 | no_direct_path |
x-venom-mesh-mode: direct with no reachable endpoint |
| 503 | session_daemon_unavailable |
a remote venom serve session's daemon is gone; resume the session |
| 504 | dispatch_timeout |
no daemon acked within the 30 s budget |
On the direct path the daemon itself refuses a bad ticket with 401 and one of ticket_invalid, ticket_expired, ticket_audience, ticket_replayed or ticket_team_not_accepted.
Related
- Connect a daemon: enroll a machine and set its endpoints.
- Daemon API reference: every route a daemon calls.
- Gears: route profiles such as
daemons-firstchoose between daemons and cloud providers per request.