Skip to content
Docs menu

Chat, responses, embeddings and models

Call chat completions, completions, the Responses API (including background responses) and embeddings through the orchestrator, and read what the model list tells you.

View .md

These are the OpenAI endpoints most clients use. Requests and answers are OpenAI's; the orchestrator rewrites model, applies each upstream kind's parameter rules and relays the answer, streamed or not. The examples assume export OPENAI_BASE_URL=http://127.0.0.1:8787/v1 and a client key in OPENAI_API_KEY, which the OpenAI SDKs read.

Chat completions

POST /v1/chat/completions, streaming (SSE) and non-streaming, with tools, images and reasoning.

curl -N "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" -H "x-venom-session: demo-1" \
  -d '{"model":"venom-default","stream":true,"messages":[{"role":"user","content":"Write a haiku about Go"}]}'

What the orchestrator does on the way:

  • Removes parameters the target doesn't take and renames the ones it spells differently (max_completion_tokens → max_tokens on Ollama, seed → random_seed on Mistral), and maps reasoning_effort to the nearest level the model accepts.
  • Normalises reasoning to reasoning_content in answers, and sends earlier turns' reasoning back in the field the server reads (or drops it where it reads none).
  • Adds stream_options.include_usage where needed so a stream ends with usage; a client that didn't ask for usage doesn't get the extra chunk.
  • Repairs malformed tool-call arguments some local models emit (a single-quoted or trailing-comma JSON object), counted in venom_tool_call_repairs_total.
  • A request with an image skips targets known to be text-only, or goes to the route's images route.

A stream ends at its terminal event (data: [DONE]); if an upstream fails after the first byte, the stream ends with an error event in OpenAI's shape and the client retries.

Completions and embeddings

POST /v1/completions and POST /v1/embeddings are routed the same way and passed through after the model is rewritten. On a mistral target, /v1/completions with a suffix becomes Codestral's fill-in-the-middle call.

from openai import OpenAI

client = OpenAI()  # OPENAI_BASE_URL and OPENAI_API_KEY from the environment
vectors = client.embeddings.create(model="venom-embed", input=["def add(a, b):", "class User:"])
print(len(vectors.data[0].embedding))

Embeddings take input as a string or an array, dimensions and extension fields such as input_type. encoding_format: "base64" (the SDKs' default) works against every kind: where the upstream answers with floats, the orchestrator encodes them as OpenAI does. Bedrock serves Titan Text Embeddings V2 (one input per request) and Cohere Embed. The shipped venom-embed alias has only a local Ollama target, so code never leaves the machine for an embedding.

Responses API

POST /v1/responses, streaming and non-streaming. To a target whose capabilities.responses is true (OpenAI by default) it is passed through. To every other target it is translated to chat completions and the answer translated back into a Response object or Responses SSE events (response.created, response.output_text.delta, response.function_call_arguments.delta, …, response.completed), so a Responses client works against Ollama or vLLM off the grid. Translated answers carry x-venom-translated: chat_completions.

import OpenAI from "openai";

const client = new OpenAI(); // OPENAI_BASE_URL and OPENAI_API_KEY from the environment
const res = await client.responses.create({ model: "venom-default", input: "Name three prime numbers" });
console.log(res.output_text);

Translated: input (text, messages, function calls and outputs, reasoning items, input_image URLs), instructions, function tools, tool_choice, max_output_tokens, temperature, top_p, parallel_tool_calls, text.format, reasoning.effort, user, safety_identifier, prompt_cache_key, service_tier; metadata is echoed but not sent. The translation is stateless: previous_response_id and conversation work only on native targets, and with none left the answer is 400 stateful_request_unsupported. Hosted tools (web_search, file_search), item_reference inputs, input_file parts and stored prompts are 400 unsupported_request unless a native target remains.

Background responses

background: true returns at once with a queued Response and runs the request on its own.

Endpoint What it does
GET /v1/responses/{id} The response this key created. On a native target it is read from the provider; on a translated one it is the stored object, and ?stream=true replays its events (after starting_after=<sequence_number>) and follows a running one.
POST /v1/responses/{id}/cancel Cancels it; a completed translated one is 400 invalid_state.
DELETE /v1/responses/{id} Cancels a running one and deletes it.

Another key's id, or an unknown one, is 404 not_found. A native background response is billed once, at the usage the provider reports when it ends (polled for up to 6 hours); a translated one is billed when its run ends, and its final Response is kept for 24 hours.

Models

GET /v1/models lists every route alias plus the concrete upstream/model ids that can be served; GET /v1/models/{id} is one entry (the / of an upstream/model id may be sent raw or as %2F; unknown is 404 model_not_found). A key with allowed_models sees only those. Entries carry non-standard fields clients can use:

Field Meaning
context_window Tokens a request may use. For an alias, the smallest window among the targets that can serve now, so it follows upstream health.
capabilities vision (true only when every reachable target takes images), and tts / stt on speech models.
max_inflight How many requests to this id the orchestrator runs at once (omitted when unbounded).
endpoints Set on models that serve something other than chat: ["/v1/embeddings"], ["/v1/rerank"], ["/v1/systemone"], ["/v1/audio/transcriptions"], ["/v1/audio/speech"], ["/v1/images/generations"]. Chat clients skip these.
unavailable On an alias, the targets skipped because their upstream lacks the model, each with reason and fix.
cloud_targets On an alias, the enabled targets that are not local: where prompts can go.

The list object also has a top-level profiles array with the configured route profile names.

Prewarm

POST /v1/prewarm loads the local models a set of ids routes to before the first turn, so a session doesn't wait for a cold load. The venom CLI sends it when a session starts on a gear or switches gear.

curl -s "$OPENAI_BASE_URL/prewarm" -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" -d '{"models":["venom-default","venom-fast"],"keep_alive_s":3600}'

Each id (1 to 8, else 400 invalid_body) is resolved to the first target its route would use. Ollama targets are loaded only when they fit beside what is loaded, so a prewarm never evicts a model; the answer lists each with state loaded, warm (already loaded, keep-alive renewed), skipped (doesn't fit, with the figures), not_local (a cloud target) or not_served. keep_alive_s defaults to 1 hour, at most 12.

Request headers

Header What it does
x-venom-session Keeps a session on one target for cache reuse; <session>:<kind> marks a side request.
x-venom-parent-session Rolls a subagent's spend up under its parent; also inherits its experiment variant.
x-venom-profile Picks a route profile.
x-venom-role, x-venom-gear The spend lane (workers, smart, vision, browser, classifier, summary, reflex, …) and gear, recorded in the request log.
x-venom-intent Meters the request against an intent's spend cap (Keys, budgets and limits).
x-venom-service-tier, x-venom-urgency Provider tier and batch requests (Batches, files and service tiers).
x-venom-cache-ttl long asks for the model's longer prompt-cache lifetime (cache_retention).
x-venom-dedup off opts out of request deduplication.
Idempotency-Key Collapses retries of the same request and replays the completed answer.

Response headers

Every answer carries x-venom-request-id and x-request-id (the upstream's own, when it sent one). Committed answers carry x-venom-upstream and x-venom-model before the first byte. Depending on the request: x-venom-fallback, x-venom-base-model, x-venom-translated, x-venom-variant, x-venom-profile, x-venom-response-cache, x-venom-dedup, x-venom-queued-ms, x-venom-budget-warning, x-venom-guardrail, x-venom-context-cost-usd. Upstream headers pass by allowlist only (content type and length, request ids, retry-after, x-ratelimit-*, openai-processing-ms, openai-version); cookies, account ids and server banners never reach the client.