Skip to content
Docs menu

Inference API

Call models through Venom Cloud with any OpenAI SDK, stream answers, run background responses, list models, resume a dropped stream and route mesh-aware clients to a daemon.

View .md

These routes are the OpenAI-compatible part of Venom Cloud: the same paths and bodies as the self-hosted orchestrator, served at https://cloud.usevenom.dev/v1. Each request is routed to one of your daemons or to the cloud's own providers, under the key's models, limits, budgets and your org's filters. Every route here takes a client key (Authorization: Bearer vck_…, see Credentials).

Quick start

export VENOM_CLOUD=https://cloud.usevenom.dev
export VENOM_KEY=vck_...

# A streamed chat completion
curl -sN "$VENOM_CLOUD/v1/chat/completions" \
  -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -H "x-venom-session: support-ticket-4812" \
  -d '{"model": "venom-auto", "stream": true,
       "messages": [{"role": "user", "content": "Summarise this ticket in one line: ..."}]}'
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://cloud.usevenom.dev/v1", apiKey: process.env.VENOM_KEY });

const stream = await client.chat.completions.create(
  { model: "venom-auto", stream: true, messages: [{ role: "user", content: "Summarise this ticket: ..." }] },
  { headers: { "x-venom-session": "support-ticket-4812", "x-venom-project": "support-bot" } },
);
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
import os
from openai import OpenAI

client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
emb = client.embeddings.create(model="text-embedding-3-small", input=["invoice overdue", "payment received"])
print(len(emb.data[0].embedding))

venom-auto picks a cheap, balanced or premium model per request from your org's tier map; any model id that GET /v1/models lists works too.

Routes

Route What it does
POST /v1/chat/completions Chat completions, streamed ("stream": true, SSE) or not. Tools, vision and JSON mode pass through to the serving model.
POST /v1/completions Legacy text completions.
POST /v1/responses The Responses API. With "background": true it starts a response you read later (Background responses).
GET /v1/responses/{id} A background response as it stands; ?stream=true streams its events, &starting_after=N from after sequence number N.
POST /v1/responses/{id}/cancel Stops a queued or running background response. Idempotent: an ended one comes back as it ended.
DELETE /v1/responses/{id} Deletes a background response: {"id", "object": "response", "deleted": true}. A running one is cancelled and billed first.
POST /v1/embeddings Embeddings. Never streamed.
POST /v1/rerank Rerank documents against a query (Rerank). Never streamed.
POST /v1/audio/speech, GET /v1/audio/voices, POST /v1/audio/transcriptions, POST /v1/audio/translations Text to speech, voices and transcription on your provider keys (Voice API).
POST /v1/systemone Typed decisions in TypeSafe's System One wire format (the venom-decide alias). Never streamed: "stream": true is 400 invalid_decision_request.
GET /v1/models The models this key can use right now (Models).
GET /v1/models/{id} One of them; ids with / work as is. Unknown or not served now: 404 model_not_found.
GET /v1/deprecations The org's gears that name a model its provider retires soon or has retired (Deprecations).
GET /v1/routing How this key's requests were routed: per model, the daemons that served, shares, errors, latency, fallbacks (Routing report).
POST /v1/prewarm Ask the daemons of the key's key group to load a session's models ahead of the first request (Prewarm).
POST /v1/mesh/route For mesh-aware clients: pick a daemon and get a session ticket to call it directly (Mesh routing).
GET /v1/mesh/jobs/{job_id}/stream Resume a relayed stream that broke (Resuming a stream).
GET /v1/mesh/daemons The daemons this key can use: id, name, region, models, state, load. Never their endpoints.

The generated reference with every route's method, auth and a curl line is at /docs/api/cloud. The request bodies are OpenAI's; the self-hosted Chat and Responses page documents the same surface.

How a request is served

Every inference request goes through the same steps before any model sees it: the key (expiry, pause, model allowlist), the plan's request cap and the org's spend cap, the session guard, rate limits (RPM, TPM), the org's prompt filters, venom-auto, spend guards, model guardrails, then routing. A request refused at any step costs nothing.

Routing then chooses one of three paths. The response header x-venom-served-by and x-venom-mesh-mode say which one ran.

Path When What the client sees
Relay x-venom-mesh-mode: relay, or auto when no daemon endpoint is reachable and closer; always when the org has response filters in scope and for batch urgency the cloud streams the daemon's answer byte for byte; x-venom-mesh-mode: relay
Direct x-venom-mesh-mode: direct, or auto when a daemon's own endpoint is closer than the relay 307 to <daemon endpoint><path>?venom_ticket=vtk.…, a single-use ticket valid 60 seconds; the client re-sends the same body there
Cloud providers no daemon serves the model, or the org's routing prefers the cloud the cloud calls the provider itself; x-venom-served-by: cloud, x-venom-daemon: cloud
  • x-venom-mesh-mode takes auto (default), direct or relay; anything else is 400 invalid_request. direct with no reachable endpoint is 503 no_direct_path. Declare private networks the client can reach with x-venom-networks: tailnet:<name>,lan:<label>.
  • openai-python and fetch-based clients follow the 307. A client that does not follow redirects should send x-venom-mesh-mode: relay.
  • Relayed bodies are at most 4 MiB (413 request_too_large); larger ones need the direct path.
  • A daemon that fails before the first byte (429 or 5xx) is retried on the next candidate. After the first byte nothing is retried: a broken stream ends with an SSE error event you can resume.
  • Requests of one x-venom-session stick to one daemon, so its prompt cache stays warm. Pin a daemon with x-venom-daemon: <name or id>.

All request and response headers are listed in Request and response headers.

Streaming

"stream": true on chat completions, completions and responses gives the provider's SSE stream unchanged. Nothing is sent before the daemon's first event, so an error before then is a normal HTTP error with a JSON body. After the first byte the status is already 200, so a failure arrives as a final event:

data: {"error": {"message": "The relayed stream reached the function's time limit. Resume with GET /v1/mesh/jobs/{job_id}/stream?after=<last_seq>.", "type": "api_error", "code": "stream_time_limit", "job_id": "0193…", "last_seq": 412}}
Event code Meaning What to do
daemon_lost the daemon died mid-stream resume within 120 seconds
stream_time_limit the stream reached the cloud function's time limit (800 seconds on the hosted tier) resume
daemon_error the daemon reported an error after the first byte retry the request
mesh_unavailable the relay lost its database retry later

If the client hangs up, the job is cancelled (unless an identical request joined it, below).

Resuming a stream

GET /v1/mesh/jobs/{job_id}/stream?after=<seq> replays a relayed job's events after sequence number seq, then follows it live. Take job_id from the response's x-venom-job header (or the error event) and seq from the error event's last_seq.

curl -sN "$VENOM_CLOUD/v1/mesh/jobs/0193a4c2-.../stream?after=412" -H "Authorization: Bearer $VENOM_KEY"
Parameter Type Default Rule
after integer 0 the last seq you have; 0 replays from the start. Not an integer ≥ 0: 400 invalid_request
  • Only the key that made the request can resume it; any other key gets 404 not_found.
  • It works while the job runs and for 120 seconds after it ends; later it is 410 job_gone.
  • The org's response filters apply to the resumed events as they did to the first stream.
  • Plain OpenAI SDKs can't resume on their own; the venom harness and your own SSE reader can.

Retries, idempotency and deduplication

  • Idempotency-Key (at most 255 characters, else 400 invalid_request): the same key and API key within 24 hours re-attaches to a running relayed job and replays it from the start. A finished one answers 409 idempotency_conflict with its x-venom-job (content is not kept, so it can't be replayed).
  • Automatic deduplication on the relay path: a deterministic request (temperature: 0 set explicitly), any /v1/embeddings or /v1/rerank request, from the same API key with the same body while an identical one is still running, joins that job instead of starting a new one. The answer carries x-venom-dedup: joined and is billed once. Send x-venom-dedup: off to opt out; an Idempotency-Key also turns it off. Direct requests are deduplicated by the daemon's orchestrator instead.

Background responses

POST /v1/responses with "background": true returns at once and keeps running. Background responses always run on the cloud's own providers (a daemon's response would leave the cloud no way to answer the calls that follow). A model no cloud provider serves for this key is 400 background_unsupported with param: "background", and so is any background request while a response filter is in scope for it.

import time
from openai import OpenAI

client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
r = client.responses.create(model="gpt-5-mini", input="Draft a payment reminder for invoice 1042.", background=True)
while r.status in ("queued", "in_progress"):
    time.sleep(2)
    r = client.responses.retrieve(r.id)
print(r.output_text)
  • With a provider that speaks the Responses API natively, the request goes through as sent, and GET, cancel and DELETE go to that provider with the same key. Leaving a stream does not stop the response.
  • With a chat-only provider the cloud answers status: "queued" with its own id and runs the request as streamed chat completions after the reply. It must finish within the function's limit less 30 seconds (770 seconds on the hosted tier), else it fails with background_timeout.
  • GET /v1/responses/{id}?stream=true&starting_after=N streams its events. A stream that would outlive the function closes early: reconnect with the last sequence number you saw.
  • Only the key's key group can read, cancel or delete it; any other id is 404 not_found.
  • Billing: the budget is reserved at the start and the response is billed once when it is seen to end (your GET, a stream, a cancel, a delete, or the cloud's own poll). One still unresolved after 6 hours is billed at the estimate. Background requests show in usage and the log explorer as is:background.
  • Content: a native response's row holds no content and is kept 7 days after it is billed; an emulated response's content is deleted 24 hours after it ends.

Rerank

The body is the self-hosted orchestrator's (Rerank) and is checked before anything is spent: a bad one is 400 invalid_rerank_request.

curl -s "$VENOM_CLOUD/v1/rerank" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"model": "rerank-2", "query": "overdue invoice", "top_n": 2,
       "documents": ["Invoice 1042 is 30 days overdue", "Welcome to the team", {"text": "Payment reminder sent"}]}'
Field Type Default Rule
model string required a rerank model the key may use
query string required not empty
documents array required 1 to 1,000 entries, each a string or {"text": "..."}
top_n (or top_k) integer all at least 1
return_documents boolean false include each document's text in the results
max_tokens_per_doc number provider default positive
instruct string none an instruction for models that take one

The answer's results are best first: {index, relevance_score, document?}.

Models

GET /v1/models answers {"object": "list", "data": [...]}: one entry per model id that at least one live daemon the key may use serves, or that the cloud's own providers serve for the key. A model with neither is left out, so the list is what can be served now. Entries keep the orchestrator's fields and add a mesh object:

Field Value
id, object: "model", owned_by as OpenAI
context_window the smallest value among the serving daemons
capabilities.vision true only if every serving daemon says so
max_inflight the sum over the serving daemons; left out if any is unbounded
cloud_targets the union of the serving daemons' cloud targets
mesh.daemons how many daemons serve it (0 when only the cloud does)
mesh.regions their regions
mesh.local_only true when no serving daemon has cloud targets and the cloud does not serve it
mesh.cloud true when the cloud's own providers serve it

venom-auto is listed when the key may use it and at least one of the org's tier models is listed. The list is filtered by the key's allowed_models and denied_models.

Deprecations

GET /v1/deprecations lists the models the org's gears name that are retiring (within the org's warning window, 30 days by default) or already gone. venom doctor calls it once and warns.

{"object": "list", "warn_days": 30, "default_gear": "eco", "gears": 4,
 "data": [{"model": "gpt-4o-2024-05-13", "retires": "2026-12-01", "days_left": 28, "state": "soon",
           "replacement": "gpt-4.1", "gears": ["deep"]}]}

state is soon, retired or gone. Admins see every use and switch gears in one step on the console's Model lifecycle page.

Routing report

GET /v1/routing is the cloud's version of the orchestrator's venom.routing view, for this key only: per model alias, the requests, the daemons that served them (targets[].upstream is the daemon's name, model the id it served), shares, errors, tokens, cost, latency and TTFT percentiles, and recent fallbacks with their reason.

Parameter Type Default Rule
window duration 24h <n>m, <n>h or <n>d, from 1 minute to 31 days
recent integer 10 recent fallbacks per alias, 0 to 100

The cloud's own quality probes are left out.

Prewarm

POST /v1/prewarm asks the live daemons of the key's key group that advertise the given models to load them with their next heartbeat answer. Nothing is loaded by the call itself. The venom harness sends it with a session's gear models.

curl -s "$VENOM_CLOUD/v1/prewarm" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"models": ["qwen3-coder:30b"], "keep_alive_s": 1800}'
Field Type Default Rule
models string[] required 1 to 8 model ids, each at most 200 characters
keep_alive_s integer 3600 how long to keep them loaded, at most 43,200 (12 hours)

The answer lists each model as {"model", "state": "requested", "daemons": n}, or {"model", "state": "not_served", "detail"} when no live daemon of the key group advertises it or the key may not call it.

Mesh routing for mesh-aware clients

POST /v1/mesh/route returns a daemon and a session ticket, so a client can send a whole session straight to the daemon. It never sees a prompt.

curl -s "$VENOM_CLOUD/v1/mesh/route" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen3-coder:30b", "session": "s-123", "mode": "auto",
       "needs": {"vision": false, "tools": true, "min_context": 64000}, "networks": ["tailnet:acme.ts.net"]}'
Field Type Default Meaning
model string required the model
session string none the session id, for affinity and the ticket
needs object {} vision, tools (booleans), min_context (tokens)
networks string[] [] networks the client can reach, for private endpoints
mode string auto auto, direct or relay
{"object": "venom.mesh.route", "mode": "direct",
 "daemon": {"id": "0193…", "name": "gpu-sg-1", "region": "ap-southeast-1"},
 "endpoint": "https://gpu-sg-1.example.com/v1", "ticket": "vtk.…", "expires_at": "2026-11-03T10:10:00Z",
 "alternates": [{"daemon": {"id": "…", "name": "…", "region": "…"}, "endpoint": "…", "ticket": "vtk.…"}]}
  • Send Authorization: Bearer vtk.… to endpoint. A session ticket lasts 10 minutes; refresh at 80% of expires_at. At most 2 alternates are given for your own failover.
  • mode: "relay" (or served_by: "cloud") means use the cloud's /v1: endpoint is the cloud, ticket and expires_at are null. An org whose prompt or response filters could act on the session always gets relay, whatever you asked for, so filters can't be bypassed.
  • The key's model access, guardrails, the org's spend cap and its spend guards are checked here. A key whose hard budget is already spent gets 429 budget_exceeded.
  • The daemon reports usage itself; the client reports nothing back.

GET /v1/mesh/daemons answers {"object": "list", "data": [{"id", "name", "region", "models", "state", "load": {"inflight", "queued", "slots"}}]} with state one of active, suspect (no heartbeat for 20 seconds), offline or draining. Daemons shared with the key's key group list only the models shared with it.

Errors

The common codes are in Errors. These are specific to inference:

Status code Meaning
400 invalid_request not a JSON object, no model, a bad x-venom-mesh-mode, an Idempotency-Key over 255 characters
400 invalid_rerank_request, invalid_decision_request a bad rerank body; a streamed /v1/systemone
400 background_unsupported no cloud provider for a background response, or a response filter in scope
403 model_not_allowed the key, its role or a guardrail does not allow the model (reason says which)
403 guardrail_blocked an org, key group or member guardrail refused the request
403 profile_forbidden x-venom-profile admits nothing the org's routing allows for the model
403 unpriced_model the model has no price, so a budget can't limit it; set one on the console's Pricing page
404 model_not_found no daemon or provider key serves the model
409 idempotency_conflict that Idempotency-Key already finished
410 job_gone a resume more than 120 seconds after the job ended
429 session_paused the org's session guard paused this session (x-should-retry: false)
502 daemon_error every candidate failed before the first byte
503 no_daemon_available, no_direct_path nothing can serve now (reasons says why); no direct endpoint is reachable
504 dispatch_timeout no daemon accepted the job in time