Skip to content
Docs menu

How the private mesh works

How Venom Cloud routes a key group's requests to its own daemons, directly with a 307 and a ticket or relayed through the cloud, and what each side sees, signs and keeps.

View .md

Venom Cloud is a control plane: stateless functions plus one Postgres database. It authenticates the client, applies rate limits and budgets, picks a daemon, and either sends the client straight to that daemon or carries the stream itself. The daemon, venom's Go orchestrator, is the data plane. The normative contract is docs/cloud-mesh.md in the source tree, and the decision record is ADR 0017; this page describes what an operator or a client author needs from them.

The parties

Party What it is Talks to
Client the venom CLI or any OpenAI SDK, with a vck_… client key the cloud's /v1; on the direct path, also a daemon's /v1 with a ticket
Cloud Next.js functions (on Vercel or next start) and Postgres clients; daemons only through the requests daemons make to it
Daemon the orchestrator in mesh mode, with its own engines and keys the cloud's daemon API, outbound only (/api/mesh/v1/*, see the daemon API reference)

A daemon belongs to the key group whose join token enrolled it. A key group holds API keys, daemons, policies and GPU grants, not people; the API and the daemon protocol call it a team (team_id, accept.teams, VENOM_MESH_ACCEPT_TEAMS). Other key groups of the same tenant can use a daemon through a share (console: Daemons, the daemon, Shares; optionally only some models, with a weight) or a GPU grant (all of a key group's daemons, with a daily token quota and an end date). There is no sharing across tenants; the public mesh is the separate, verified pool for that.

Identities and tokens

Thing Format Lifetime
Client key vck_… until revoked
Join token vjt_…, a number of uses default 1 hour, at most 7 days; 1 use from the console
Daemon key pair Ed25519; the private key is $VENOM_DATA_DIR/mesh/daemon.key (mode 0600) and never leaves the machine until rotated (orchestrator mesh rotate)
Daemon request JWT compact JWS, alg: EdDSA, kid = daemon id, bound to the method, path and body hash at most 60 s, with ±30 s of clock skew
Cloud signing keys Ed25519, in the cloud's VENOM_MESH_SIGNING_KEYS; published at GET /api/mesh/v1/jwks rotated by adding a key first
Ticket vtk. + a JWS the cloud signs 60 s and single use per request; 10 min for a session

The join token is spent once, at enrollment: the daemon sends it with its new public key and gets back a daemon id. From then on every daemon call is signed with the private key. The cloud stores only the public key, so nothing it holds lets someone act as the daemon.

Two paths for a request

Every client request first reaches the cloud's /v1. After authentication, limits and a budget pre-charge, the router picks a daemon and a path:

Path How it works Added time to first token
Direct The cloud answers 307 Location: <daemon endpoint><path>?venom_ticket=vtk.…. The client re-sends the same body to the daemon, which checks the ticket offline and streams the answer straight back. The daemon reports usage afterwards with POST /api/mesh/v1/usage. one client-to-cloud round trip plus about 5 to 20 ms of routing; nothing per token
Direct, mesh-aware The client calls POST /v1/mesh/route once per session and reuses the session ticket (Authorization: Bearer vtk.…), refreshing at 80% of its life. nothing after the first request
Relay The cloud queues a job, the daemon's open channel receives a dispatch, the daemon acks and posts its response in event batches, and the cloud streams them to the client. roughly RTT(cloud to daemon) + RTT(client to cloud) − RTT(client to daemon) + 20 to 40 ms; per token at most a 20 ms batch window

The OpenAI Python SDK and most others follow a 307 and re-send the body, so the direct path works with plain SDKs. A request over 4 MiB cannot pass through the cloud (413 request_too_large) and needs the direct path.

Which path is used

A daemon can take direct requests only when it advertises an endpoint (VENOM_MESH_ENDPOINTS) the client can reach:

Endpoint kind URL rule Reachable by
public https:// every client
tunnel https:// (for example Cloudflare Tunnel or Tailscale Funnel) every client
tailnet http:// allowed on a 100.64.0.0/10 address or a *.ts.net name; network required clients that send x-venom-networks: tailnet:<name>
lan http:// allowed; network required clients that send x-venom-networks: lan:<label>

The client has to send x-venom-networks itself. The contract names a CLI variable for it, VENOM_NETWORKS=tailnet:acme.ts.net, but the current venom CLI does not read it, so from the CLI only public and tunnel endpoints are reached directly. A daemon with no reachable endpoint is relayed.

In the default auto mode the router also compares the two paths per request: it estimates the direct round trip from the client's and the daemon's locations and the relay's cost from the cloud's measured dispatch-to-ack time, and relays when that is faster. A client far from a daemon that sits next to the cloud often gets its first byte sooner through the relay. When a location is unknown the path stays direct. The daemon's location is VENOM_MESH_GEO, or a region inferred from its time zone; it never asks a geo-IP service.

Request header Values Effect
x-venom-mesh-mode auto (default), direct, relay direct fails with 503 no_direct_path when no endpoint is reachable; explicit values are never second-guessed
x-venom-networks tailnet:<name>,lan:<label> networks the client can reach
x-venom-daemon a daemon name or id pins the request to that daemon, if the key may use it; a down daemon then gives 503
x-venom-session, x-venom-parent-session a session id keeps a session on the same daemon so its prefix cache stays warm

Response headers say what happened: x-venom-mesh-mode (direct or relay), x-venom-daemon, x-venom-daemon-region, x-venom-job, x-venom-route-reason (affinity, p2c, argmin, pinned, session_pin, cloud, hedge, failover:<reason>) and x-venom-served-by (daemon or cloud). The daemon's own x-venom-upstream and x-venom-model are passed through. An org whose prompt filters could act on a session is always relayed, so the filters see every prompt.

Picking a daemon

The router keeps only daemons that are live (heartbeat within the lease), accept the key's key group, serve the model with the needed features (vision, tools, context window), and are not held off by their circuit breaker. With a session id it picks by rendezvous hashing with bounded load, so a session stays on its daemon until that daemon is over its fair share. Without one it compares expected time to first token (network round trip, queue, recent errors) over two random candidates. When no daemon is eligible and the org has cloud provider keys configured, the cloud's own providers serve the request (x-venom-served-by: cloud).

Before the first byte reaches the client, a relayed job moves to the next candidate when the daemon does not ack within 2 s, fails with a 429 or 5xx, or loses its job lease: at most 3 attempts within 30 s. After the first byte nothing is retried; the client gets a terminal SSE error event (daemon_lost, daemon_error, stream_time_limit) and can retry or resume with GET /v1/mesh/jobs/{job_id}/stream?after=<seq> for 120 s after the end.

What stays on the daemon

  • Provider keys, engine URLs and the orchestrator's own client keys. The cloud's dispatch carries the client's JSON body and an allowlist of headers, never Authorization.
  • The private key and the per-key-group cache salt ($VENOM_DATA_DIR/mesh/cache-salt.key). Requests from key groups other than the daemon's own get a cache_salt derived from it on vllm and sglang upstreams, so one key group cannot time another's prefix cache.
  • Content on the direct path. The daemon sends usage only; it sends request and response bodies (POST /api/mesh/v1/content) only when the ticket says the key's org logs content.
  • The local /v1. Its keys, budgets and request log work as before; mesh requests appear in the log with the key name mesh:<name>.

The daemon checks each ticket against the cloud's published keys (cached for an hour, refreshed on an unknown kid), that it is the ticket's audience, that the key group is in its accept set (and in VENOM_MESH_ACCEPT_TEAMS when set) and that the model is allowed. A request ticket's id is remembered until it expires, so it cannot be replayed.

Liveness

Constant Value
Heartbeat every 10 s (stretched for large fleets: live daemons ÷ 50 per second)
Suspect no heartbeat for 20 s; deprioritised, not dropped
Lease 30 s; no new work after it lapses
Channel an SSE stream the cloud closes at its function's limit minus 30 s (270 s on Vercel Hobby, 770 s on Pro); the daemon opens the next one 15 s early
Long-poll fallback GET /api/mesh/v1/poll, 25 s
Ack timeout 2 s from dispatch
Retired 14 days offline: hidden, still revocable

The timing constants live in services/nodejs-cloud/contracts/timing.json, which the cloud and the Go daemon both test against.

enrolled ──first heartbeat──▶ active ⇄ suspect ──no heartbeat 30 s──▶ offline ──14 d──▶ retired
active ──drain (console, SIGTERM)──▶ draining ──jobs done──▶ offline
any ──revoke──▶ revoked (terminal; the daemon gets 401 daemon_revoked and stops)

When the cloud is unreachable, the local /v1 keeps serving, direct tickets already issued keep working until they expire, and usage reports queue in memory (up to 10,000 or one hour). On a restart, jobs that had sent no byte are re-dispatched by the cloud.

Errors clients see

Status code Meaning
404 model_not_found no daemon serves the model, and none ever advertised it
413 request_too_large over 4 MiB through the cloud; use the direct path
502 daemon_error every attempt failed before the first byte
503 no_daemon_available none eligible; reasons lists why (offline, breaker, model, team for the key group)
503 no_direct_path x-venom-mesh-mode: direct with no reachable endpoint
503 session_daemon_unavailable a remote venom serve session's daemon is gone; resume the session
504 dispatch_timeout no daemon acked within the 30 s budget

On the direct path the daemon itself refuses a bad ticket with 401 and one of ticket_invalid, ticket_expired, ticket_audience, ticket_replayed or ticket_team_not_accepted.

  • Connect a daemon: enroll a machine and set its endpoints.
  • Daemon API reference: every route a daemon calls.
  • Gears: route profiles such as daemons-first choose between daemons and cloud providers per request.