---
title: Inference API
description: Call models through Venom Cloud with any OpenAI SDK, stream answers, run background responses, list models, resume a dropped stream and route mesh-aware clients to a daemon.
section: cloud
order: 210
audience: [builder, agent, operator]
sources: [services/nodejs-cloud/app/v1/chat/completions/route.ts, services/nodejs-cloud/app/v1/completions/route.ts, services/nodejs-cloud/app/v1/responses, services/nodejs-cloud/app/v1/embeddings/route.ts, services/nodejs-cloud/app/v1/rerank/route.ts, services/nodejs-cloud/app/v1/systemone/route.ts, services/nodejs-cloud/app/v1/models, services/nodejs-cloud/app/v1/deprecations/route.ts, services/nodejs-cloud/app/v1/mesh, services/nodejs-cloud/app/v1/routing/route.ts, services/nodejs-cloud/app/v1/prewarm/route.ts, services/nodejs-cloud/lib/relay/inference.ts, services/nodejs-cloud/lib/relay/mesh-route.ts, services/nodejs-cloud/lib/relay/stream.ts, services/nodejs-cloud/lib/models/aggregate.ts, services/nodejs-cloud/lib/providers/rerank.ts, services/nodejs-cloud/lib/prewarm/index.ts, services/nodejs-cloud/lib/lifecycle/index.ts, services/nodejs-cloud/lib/routing-view/index.ts, docs/cloud-mesh.md]
features: [cloud.v1-chat, cloud.v1-completions, cloud.v1-responses, cloud.v1-embeddings, cloud.v1-rerank, cloud.v1-models, cloud.v1-deprecations, cloud.v1-mesh, cloud.v1-routing, cloud.v1-systemone, cloud.v1-prewarm]
---

These routes are the OpenAI-compatible part of Venom Cloud: the same paths and bodies as the self-hosted
orchestrator, served at `https://cloud.usevenom.dev/v1`. Each request is routed to one of your daemons or to the
cloud's own providers, under the key's models, limits, budgets and your org's filters. Every route here takes a
client key (`Authorization: Bearer vck_…`, see [Credentials](/docs/cloud/api-overview.md#credentials)).

## Quick start {#quick-start}

```bash
export VENOM_CLOUD=https://cloud.usevenom.dev
export VENOM_KEY=vck_...

# A streamed chat completion
curl -sN "$VENOM_CLOUD/v1/chat/completions" \
  -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -H "x-venom-session: support-ticket-4812" \
  -d '{"model": "venom-auto", "stream": true,
       "messages": [{"role": "user", "content": "Summarise this ticket in one line: ..."}]}'
```

```ts
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://cloud.usevenom.dev/v1", apiKey: process.env.VENOM_KEY });

const stream = await client.chat.completions.create(
  { model: "venom-auto", stream: true, messages: [{ role: "user", content: "Summarise this ticket: ..." }] },
  { headers: { "x-venom-session": "support-ticket-4812", "x-venom-project": "support-bot" } },
);
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
```

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
emb = client.embeddings.create(model="text-embedding-3-small", input=["invoice overdue", "payment received"])
print(len(emb.data[0].embedding))
```

`venom-auto` picks a cheap, balanced or premium model per request from your org's tier map; any model id that
`GET /v1/models` lists works too.

## Routes {#routes}

| Route | What it does |
|---|---|
| `POST /v1/chat/completions` | Chat completions, streamed (`"stream": true`, SSE) or not. Tools, vision and JSON mode pass through to the serving model. |
| `POST /v1/completions` | Legacy text completions. |
| `POST /v1/responses` | The Responses API. With `"background": true` it starts a response you read later ([Background responses](#background)). |
| `GET /v1/responses/{id}` | A background response as it stands; `?stream=true` streams its events, `&starting_after=N` from after sequence number N. |
| `POST /v1/responses/{id}/cancel` | Stops a queued or running background response. Idempotent: an ended one comes back as it ended. |
| `DELETE /v1/responses/{id}` | Deletes a background response: `{"id", "object": "response", "deleted": true}`. A running one is cancelled and billed first. |
| `POST /v1/embeddings` | Embeddings. Never streamed. |
| `POST /v1/rerank` | Rerank documents against a query ([Rerank](#rerank)). Never streamed. |
| `POST /v1/audio/speech`, `GET /v1/audio/voices`, `POST /v1/audio/transcriptions`, `POST /v1/audio/translations` | Text to speech, voices and transcription on your provider keys ([Voice API](/docs/cloud/api-voice.md)). |
| `POST /v1/systemone` | Typed decisions in TypeSafe's System One wire format (the `venom-decide` alias). Never streamed: `"stream": true` is `400 invalid_decision_request`. |
| `GET /v1/models` | The models this key can use right now ([Models](#models)). |
| `GET /v1/models/{id}` | One of them; ids with `/` work as is. Unknown or not served now: `404 model_not_found`. |
| `GET /v1/deprecations` | The org's gears that name a model its provider retires soon or has retired ([Deprecations](#deprecations)). |
| `GET /v1/routing` | How this key's requests were routed: per model, the daemons that served, shares, errors, latency, fallbacks ([Routing report](#routing)). |
| `POST /v1/prewarm` | Ask the daemons of the key's key group to load a session's models ahead of the first request ([Prewarm](#prewarm)). |
| `POST /v1/mesh/route` | For mesh-aware clients: pick a daemon and get a session ticket to call it directly ([Mesh routing](#mesh-route)). |
| `GET /v1/mesh/jobs/{job_id}/stream` | Resume a relayed stream that broke ([Resuming a stream](#resume)). |
| `GET /v1/mesh/daemons` | The daemons this key can use: id, name, region, models, state, load. Never their endpoints. |

The generated reference with every route's method, auth and a curl line is at `/docs/api/cloud`. The request
bodies are OpenAI's; the self-hosted [Chat and Responses](/docs/orchestrator/chat-and-responses.md) page documents
the same surface.

## How a request is served {#paths}

Every inference request goes through the same steps before any model sees it: the key (expiry, pause, model
allowlist), the plan's request cap and the org's spend cap, the session guard, rate limits (RPM, TPM), the org's
prompt filters, `venom-auto`, spend guards, model guardrails, then routing. A request refused at any step costs
nothing.

Routing then chooses one of three paths. The response header `x-venom-served-by` and `x-venom-mesh-mode` say
which one ran.

| Path | When | What the client sees |
|---|---|---|
| Relay | `x-venom-mesh-mode: relay`, or `auto` when no daemon endpoint is reachable and closer; always when the org has response filters in scope and for `batch` urgency | the cloud streams the daemon's answer byte for byte; `x-venom-mesh-mode: relay` |
| Direct | `x-venom-mesh-mode: direct`, or `auto` when a daemon's own endpoint is closer than the relay | `307` to `<daemon endpoint><path>?venom_ticket=vtk.…`, a single-use ticket valid 60 seconds; the client re-sends the same body there |
| Cloud providers | no daemon serves the model, or the org's routing prefers the cloud | the cloud calls the provider itself; `x-venom-served-by: cloud`, `x-venom-daemon: cloud` |

- `x-venom-mesh-mode` takes `auto` (default), `direct` or `relay`; anything else is `400 invalid_request`.
  `direct` with no reachable endpoint is `503 no_direct_path`. Declare private networks the client can reach with
  `x-venom-networks: tailnet:<name>,lan:<label>`.
- openai-python and fetch-based clients follow the `307`. A client that does not follow redirects should send
  `x-venom-mesh-mode: relay`.
- Relayed bodies are at most 4 MiB (`413 request_too_large`); larger ones need the direct path.
- A daemon that fails before the first byte (429 or 5xx) is retried on the next candidate. After the first byte
  nothing is retried: a broken stream ends with an SSE error event you can [resume](#resume).
- Requests of one `x-venom-session` stick to one daemon, so its prompt cache stays warm. Pin a daemon with
  `x-venom-daemon: <name or id>`.

All request and response headers are listed in [Request and response headers](/docs/cloud/api-overview.md#headers).

## Streaming {#streaming}

`"stream": true` on chat completions, completions and responses gives the provider's SSE stream unchanged.
Nothing is sent before the daemon's first event, so an error before then is a normal HTTP error with a JSON
body. After the first byte the status is already `200`, so a failure arrives as a final event:

```text
data: {"error": {"message": "The relayed stream reached the function's time limit. Resume with GET /v1/mesh/jobs/{job_id}/stream?after=<last_seq>.", "type": "api_error", "code": "stream_time_limit", "job_id": "0193…", "last_seq": 412}}
```

| Event `code` | Meaning | What to do |
|---|---|---|
| `daemon_lost` | the daemon died mid-stream | resume within 120 seconds |
| `stream_time_limit` | the stream reached the cloud function's time limit (800 seconds on the hosted tier) | resume |
| `daemon_error` | the daemon reported an error after the first byte | retry the request |
| `mesh_unavailable` | the relay lost its database | retry later |

If the client hangs up, the job is cancelled (unless an identical request joined it, below).

## Resuming a stream {#resume}

`GET /v1/mesh/jobs/{job_id}/stream?after=<seq>` replays a relayed job's events after sequence number `seq`, then
follows it live. Take `job_id` from the response's `x-venom-job` header (or the error event) and `seq` from the
error event's `last_seq`.

```bash
curl -sN "$VENOM_CLOUD/v1/mesh/jobs/0193a4c2-.../stream?after=412" -H "Authorization: Bearer $VENOM_KEY"
```

| Parameter | Type | Default | Rule |
|---|---|---|---|
| `after` | integer | `0` | the last seq you have; `0` replays from the start. Not an integer ≥ 0: `400 invalid_request` |

- Only the key that made the request can resume it; any other key gets `404 not_found`.
- It works while the job runs and for 120 seconds after it ends; later it is `410 job_gone`.
- The org's response filters apply to the resumed events as they did to the first stream.
- Plain OpenAI SDKs can't resume on their own; the venom harness and your own SSE reader can.

## Retries, idempotency and deduplication {#idempotency}

- **`Idempotency-Key`** (at most 255 characters, else `400 invalid_request`): the same key and API key within 24
  hours re-attaches to a running relayed job and replays it from the start. A finished one answers
  `409 idempotency_conflict` with its `x-venom-job` (content is not kept, so it can't be replayed).
- **Automatic deduplication** on the relay path: a deterministic request (`temperature: 0` set explicitly),
  any `/v1/embeddings` or `/v1/rerank` request, from the same API key with the same body while an identical one
  is still running, joins that job instead of starting a new one. The answer carries `x-venom-dedup: joined` and
  is billed once. Send `x-venom-dedup: off` to opt out; an `Idempotency-Key` also turns it off. Direct requests
  are deduplicated by the daemon's orchestrator instead.

## Background responses {#background}

`POST /v1/responses` with `"background": true` returns at once and keeps running. Background responses always
run on the cloud's own providers (a daemon's response would leave the cloud no way to answer the calls that
follow). A model no cloud provider serves for this key is `400 background_unsupported` with
`param: "background"`, and so is any background request while a response filter is in scope for it.

```python
import time
from openai import OpenAI

client = OpenAI(base_url="https://cloud.usevenom.dev/v1", api_key=os.environ["VENOM_KEY"])
r = client.responses.create(model="gpt-5-mini", input="Draft a payment reminder for invoice 1042.", background=True)
while r.status in ("queued", "in_progress"):
    time.sleep(2)
    r = client.responses.retrieve(r.id)
print(r.output_text)
```

- With a provider that speaks the Responses API natively, the request goes through as sent, and `GET`,
  `cancel` and `DELETE` go to that provider with the same key. Leaving a stream does not stop the response.
- With a chat-only provider the cloud answers `status: "queued"` with its own id and runs the request as streamed
  chat completions after the reply. It must finish within the function's limit less 30 seconds (770 seconds on the
  hosted tier), else it fails with `background_timeout`.
- `GET /v1/responses/{id}?stream=true&starting_after=N` streams its events. A stream that would outlive the
  function closes early: reconnect with the last sequence number you saw.
- Only the key's key group can read, cancel or delete it; any other id is `404 not_found`.
- Billing: the budget is reserved at the start and the response is billed once when it is seen to end (your
  `GET`, a stream, a cancel, a delete, or the cloud's own poll). One still unresolved after 6 hours is billed at
  the estimate. Background requests show in usage and the log explorer as `is:background`.
- Content: a native response's row holds no content and is kept 7 days after it is billed; an emulated
  response's content is deleted 24 hours after it ends.

## Rerank {#rerank}

The body is the self-hosted orchestrator's ([Rerank](/docs/orchestrator/rerank-images-decisions.md#rerank)) and is
checked before anything is spent: a bad one is `400 invalid_rerank_request`.

```bash
curl -s "$VENOM_CLOUD/v1/rerank" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"model": "rerank-2", "query": "overdue invoice", "top_n": 2,
       "documents": ["Invoice 1042 is 30 days overdue", "Welcome to the team", {"text": "Payment reminder sent"}]}'
```

| Field | Type | Default | Rule |
|---|---|---|---|
| `model` | string | required | a rerank model the key may use |
| `query` | string | required | not empty |
| `documents` | array | required | 1 to 1,000 entries, each a string or `{"text": "..."}` |
| `top_n` (or `top_k`) | integer | all | at least 1 |
| `return_documents` | boolean | `false` | include each document's text in the results |
| `max_tokens_per_doc` | number | provider default | positive |
| `instruct` | string | none | an instruction for models that take one |

The answer's `results` are best first: `{index, relevance_score, document?}`.

## Models {#models}

`GET /v1/models` answers `{"object": "list", "data": [...]}`: one entry per model id that at least one live daemon
the key may use serves, or that the cloud's own providers serve for the key. A model with neither is left out, so
the list is what can be served now. Entries keep the orchestrator's fields and add a `mesh` object:

| Field | Value |
|---|---|
| `id`, `object: "model"`, `owned_by` | as OpenAI |
| `context_window` | the smallest value among the serving daemons |
| `capabilities.vision` | true only if every serving daemon says so |
| `max_inflight` | the sum over the serving daemons; left out if any is unbounded |
| `cloud_targets` | the union of the serving daemons' cloud targets |
| `mesh.daemons` | how many daemons serve it (0 when only the cloud does) |
| `mesh.regions` | their regions |
| `mesh.local_only` | true when no serving daemon has cloud targets and the cloud does not serve it |
| `mesh.cloud` | true when the cloud's own providers serve it |

`venom-auto` is listed when the key may use it and at least one of the org's tier models is listed. The list is
filtered by the key's `allowed_models` and `denied_models`.

## Deprecations {#deprecations}

`GET /v1/deprecations` lists the models the org's gears name that are retiring (within the org's warning window,
30 days by default) or already gone. `venom doctor` calls it once and warns.

```json
{"object": "list", "warn_days": 30, "default_gear": "eco", "gears": 4,
 "data": [{"model": "gpt-4o-2024-05-13", "retires": "2026-12-01", "days_left": 28, "state": "soon",
           "replacement": "gpt-4.1", "gears": ["deep"]}]}
```

`state` is `soon`, `retired` or `gone`. Admins see every use and switch gears in one step on the console's Model
lifecycle page.

## Routing report {#routing}

`GET /v1/routing` is the cloud's version of the orchestrator's `venom.routing` view, for this key only: per model
alias, the requests, the daemons that served them (`targets[].upstream` is the daemon's name, `model` the id it
served), shares, errors, tokens, cost, latency and TTFT percentiles, and recent fallbacks with their reason.

| Parameter | Type | Default | Rule |
|---|---|---|---|
| `window` | duration | `24h` | `<n>m`, `<n>h` or `<n>d`, from 1 minute to 31 days |
| `recent` | integer | `10` | recent fallbacks per alias, 0 to 100 |

The cloud's own quality probes are left out.

## Prewarm {#prewarm}

`POST /v1/prewarm` asks the live daemons of the key's key group that advertise the given models to load them with their next
heartbeat answer. Nothing is loaded by the call itself. The venom harness sends it with a session's gear models.

```bash
curl -s "$VENOM_CLOUD/v1/prewarm" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"models": ["qwen3-coder:30b"], "keep_alive_s": 1800}'
```

| Field | Type | Default | Rule |
|---|---|---|---|
| `models` | string[] | required | 1 to 8 model ids, each at most 200 characters |
| `keep_alive_s` | integer | `3600` | how long to keep them loaded, at most 43,200 (12 hours) |

The answer lists each model as `{"model", "state": "requested", "daemons": n}`, or `{"model", "state":
"not_served", "detail"}` when no live daemon of the key group advertises it or the key may not call it.

## Mesh routing for mesh-aware clients {#mesh-route}

`POST /v1/mesh/route` returns a daemon and a session ticket, so a client can send a whole session straight to the
daemon. It never sees a prompt.

```bash
curl -s "$VENOM_CLOUD/v1/mesh/route" -H "Authorization: Bearer $VENOM_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen3-coder:30b", "session": "s-123", "mode": "auto",
       "needs": {"vision": false, "tools": true, "min_context": 64000}, "networks": ["tailnet:acme.ts.net"]}'
```

| Field | Type | Default | Meaning |
|---|---|---|---|
| `model` | string | required | the model |
| `session` | string | none | the session id, for affinity and the ticket |
| `needs` | object | `{}` | `vision`, `tools` (booleans), `min_context` (tokens) |
| `networks` | string[] | `[]` | networks the client can reach, for private endpoints |
| `mode` | string | `auto` | `auto`, `direct` or `relay` |

```json
{"object": "venom.mesh.route", "mode": "direct",
 "daemon": {"id": "0193…", "name": "gpu-sg-1", "region": "ap-southeast-1"},
 "endpoint": "https://gpu-sg-1.example.com/v1", "ticket": "vtk.…", "expires_at": "2026-11-03T10:10:00Z",
 "alternates": [{"daemon": {"id": "…", "name": "…", "region": "…"}, "endpoint": "…", "ticket": "vtk.…"}]}
```

- Send `Authorization: Bearer vtk.…` to `endpoint`. A session ticket lasts 10 minutes; refresh at 80% of
  `expires_at`. At most 2 `alternates` are given for your own failover.
- `mode: "relay"` (or `served_by: "cloud"`) means use the cloud's `/v1`: `endpoint` is the cloud, `ticket` and
  `expires_at` are null. An org whose prompt or response filters could act on the session always gets `relay`,
  whatever you asked for, so filters can't be bypassed.
- The key's model access, guardrails, the org's spend cap and its spend guards are checked here. A key whose hard
  budget is already spent gets `429 budget_exceeded`.
- The daemon reports usage itself; the client reports nothing back.

`GET /v1/mesh/daemons` answers `{"object": "list", "data": [{"id", "name", "region", "models", "state", "load":
{"inflight", "queued", "slots"}}]}` with `state` one of `active`, `suspect` (no heartbeat for 20 seconds),
`offline` or `draining`. Daemons shared with the key's key group list only the models shared with it.

## Errors {#errors}

The common codes are in [Errors](/docs/cloud/api-overview.md#errors). These are specific to inference:

| Status | `code` | Meaning |
|---|---|---|
| 400 | `invalid_request` | not a JSON object, no `model`, a bad `x-venom-mesh-mode`, an `Idempotency-Key` over 255 characters |
| 400 | `invalid_rerank_request`, `invalid_decision_request` | a bad rerank body; a streamed `/v1/systemone` |
| 400 | `background_unsupported` | no cloud provider for a background response, or a response filter in scope |
| 403 | `model_not_allowed` | the key, its role or a guardrail does not allow the model (`reason` says which) |
| 403 | `guardrail_blocked` | an org, key group or member guardrail refused the request |
| 403 | `profile_forbidden` | `x-venom-profile` admits nothing the org's routing allows for the model |
| 403 | `unpriced_model` | the model has no price, so a budget can't limit it; set one on the console's Pricing page |
| 404 | `model_not_found` | no daemon or provider key serves the model |
| 409 | `idempotency_conflict` | that `Idempotency-Key` already finished |
| 410 | `job_gone` | a resume more than 120 seconds after the job ended |
| 429 | `session_paused` | the org's session guard paused this session (`x-should-retry: false`) |
| 502 | `daemon_error` | every candidate failed before the first byte |
| 503 | `no_daemon_available`, `no_direct_path` | nothing can serve now (`reasons` says why); no direct endpoint is reachable |
| 504 | `dispatch_timeout` | no daemon accepted the job in time |

## Related {#related}

- [Venom Cloud APIs overview](/docs/cloud/api-overview.md): credentials, errors, limits and headers
- [Agent and CLI API](/docs/cloud/api-agent.md): gears, usage, policy and the CLI's reports
- [Chat and Responses](/docs/orchestrator/chat-and-responses.md): the same surface, self-hosted
- [Cloud quickstart](/docs/getting-started/cloud-quickstart.md)
