Chat, responses, embeddings and models
Call chat completions, completions, the Responses API (including background responses) and embeddings through the orchestrator, and read what the model list tells you.
These are the OpenAI endpoints most clients use. Requests and answers are OpenAI's; the orchestrator rewrites model, applies each upstream kind's parameter rules and relays the answer, streamed or not. The examples assume export OPENAI_BASE_URL=http://127.0.0.1:8787/v1 and a client key in OPENAI_API_KEY, which the OpenAI SDKs read.
Chat completions
POST /v1/chat/completions, streaming (SSE) and non-streaming, with tools, images and reasoning.
curl -N "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" -H "x-venom-session: demo-1" \
-d '{"model":"venom-default","stream":true,"messages":[{"role":"user","content":"Write a haiku about Go"}]}'What the orchestrator does on the way:
- Removes parameters the target doesn't take and renames the ones it spells differently (
max_completion_tokens→max_tokenson Ollama,seed→random_seedon Mistral), and mapsreasoning_effortto the nearest level the model accepts. - Normalises reasoning to
reasoning_contentin answers, and sends earlier turns' reasoning back in the field the server reads (or drops it where it reads none). - Adds
stream_options.include_usagewhere needed so a stream ends with usage; a client that didn't ask for usage doesn't get the extra chunk. - Repairs malformed tool-call arguments some local models emit (a single-quoted or trailing-comma JSON object), counted in
venom_tool_call_repairs_total. - A request with an image skips targets known to be text-only, or goes to the route's
imagesroute.
A stream ends at its terminal event (data: [DONE]); if an upstream fails after the first byte, the stream ends with an error event in OpenAI's shape and the client retries.
Completions and embeddings
POST /v1/completions and POST /v1/embeddings are routed the same way and passed through after the model is rewritten. On a mistral target, /v1/completions with a suffix becomes Codestral's fill-in-the-middle call.
from openai import OpenAI
client = OpenAI() # OPENAI_BASE_URL and OPENAI_API_KEY from the environment
vectors = client.embeddings.create(model="venom-embed", input=["def add(a, b):", "class User:"])
print(len(vectors.data[0].embedding))Embeddings take input as a string or an array, dimensions and extension fields such as input_type. encoding_format: "base64" (the SDKs' default) works against every kind: where the upstream answers with floats, the orchestrator encodes them as OpenAI does. Bedrock serves Titan Text Embeddings V2 (one input per request) and Cohere Embed. The shipped venom-embed alias has only a local Ollama target, so code never leaves the machine for an embedding.
Responses API
POST /v1/responses, streaming and non-streaming. To a target whose capabilities.responses is true (OpenAI by default) it is passed through. To every other target it is translated to chat completions and the answer translated back into a Response object or Responses SSE events (response.created, response.output_text.delta, response.function_call_arguments.delta, …, response.completed), so a Responses client works against Ollama or vLLM off the grid. Translated answers carry x-venom-translated: chat_completions.
import OpenAI from "openai";
const client = new OpenAI(); // OPENAI_BASE_URL and OPENAI_API_KEY from the environment
const res = await client.responses.create({ model: "venom-default", input: "Name three prime numbers" });
console.log(res.output_text);Translated: input (text, messages, function calls and outputs, reasoning items, input_image URLs), instructions, function tools, tool_choice, max_output_tokens, temperature, top_p, parallel_tool_calls, text.format, reasoning.effort, user, safety_identifier, prompt_cache_key, service_tier; metadata is echoed but not sent. The translation is stateless: previous_response_id and conversation work only on native targets, and with none left the answer is 400 stateful_request_unsupported. Hosted tools (web_search, file_search), item_reference inputs, input_file parts and stored prompts are 400 unsupported_request unless a native target remains.
Background responses
background: true returns at once with a queued Response and runs the request on its own.
| Endpoint | What it does |
|---|---|
GET /v1/responses/{id} |
The response this key created. On a native target it is read from the provider; on a translated one it is the stored object, and ?stream=true replays its events (after starting_after=<sequence_number>) and follows a running one. |
POST /v1/responses/{id}/cancel |
Cancels it; a completed translated one is 400 invalid_state. |
DELETE /v1/responses/{id} |
Cancels a running one and deletes it. |
Another key's id, or an unknown one, is 404 not_found. A native background response is billed once, at the usage the provider reports when it ends (polled for up to 6 hours); a translated one is billed when its run ends, and its final Response is kept for 24 hours.
Models
GET /v1/models lists every route alias plus the concrete upstream/model ids that can be served; GET /v1/models/{id} is one entry (the / of an upstream/model id may be sent raw or as %2F; unknown is 404 model_not_found). A key with allowed_models sees only those. Entries carry non-standard fields clients can use:
| Field | Meaning |
|---|---|
context_window |
Tokens a request may use. For an alias, the smallest window among the targets that can serve now, so it follows upstream health. |
capabilities |
vision (true only when every reachable target takes images), and tts / stt on speech models. |
max_inflight |
How many requests to this id the orchestrator runs at once (omitted when unbounded). |
endpoints |
Set on models that serve something other than chat: ["/v1/embeddings"], ["/v1/rerank"], ["/v1/systemone"], ["/v1/audio/transcriptions"], ["/v1/audio/speech"], ["/v1/images/generations"]. Chat clients skip these. |
unavailable |
On an alias, the targets skipped because their upstream lacks the model, each with reason and fix. |
cloud_targets |
On an alias, the enabled targets that are not local: where prompts can go. |
The list object also has a top-level profiles array with the configured route profile names.
Prewarm
POST /v1/prewarm loads the local models a set of ids routes to before the first turn, so a session doesn't wait for a cold load. The venom CLI sends it when a session starts on a gear or switches gear.
curl -s "$OPENAI_BASE_URL/prewarm" -H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" -d '{"models":["venom-default","venom-fast"],"keep_alive_s":3600}'Each id (1 to 8, else 400 invalid_body) is resolved to the first target its route would use. Ollama targets are loaded only when they fit beside what is loaded, so a prewarm never evicts a model; the answer lists each with state loaded, warm (already loaded, keep-alive renewed), skipped (doesn't fit, with the figures), not_local (a cloud target) or not_served. keep_alive_s defaults to 1 hour, at most 12.
Request headers
| Header | What it does |
|---|---|
x-venom-session |
Keeps a session on one target for cache reuse; <session>:<kind> marks a side request. |
x-venom-parent-session |
Rolls a subagent's spend up under its parent; also inherits its experiment variant. |
x-venom-profile |
Picks a route profile. |
x-venom-role, x-venom-gear |
The spend lane (workers, smart, vision, browser, classifier, summary, reflex, …) and gear, recorded in the request log. |
x-venom-intent |
Meters the request against an intent's spend cap (Keys, budgets and limits). |
x-venom-service-tier, x-venom-urgency |
Provider tier and batch requests (Batches, files and service tiers). |
x-venom-cache-ttl |
long asks for the model's longer prompt-cache lifetime (cache_retention). |
x-venom-dedup |
off opts out of request deduplication. |
Idempotency-Key |
Collapses retries of the same request and replays the completed answer. |
Response headers
Every answer carries x-venom-request-id and x-request-id (the upstream's own, when it sent one). Committed answers carry x-venom-upstream and x-venom-model before the first byte. Depending on the request: x-venom-fallback, x-venom-base-model, x-venom-translated, x-venom-variant, x-venom-profile, x-venom-response-cache, x-venom-dedup, x-venom-queued-ms, x-venom-budget-warning, x-venom-guardrail, x-venom-context-cost-usd. Upstream headers pass by allowlist only (content type and length, request ids, retry-after, x-ratelimit-*, openai-processing-ms, openai-version); cookies, account ids and server banners never reach the client.
Related
- Routes and strategies for fallback reasons, Keys, budgets and limits, Batches, files and service tiers.