---
title: Batches, files and service tiers
description: Run offline work with OpenAI's Batch API, turn single requests into batch requests at batch prices, upload files, and ask for flex or priority tiers.
section: orchestrator
order: 80
audience: [builder, operator]
sources: [docs/architecture.md, services/go-orchestrator/AGENTS.md, services/go-orchestrator/internal/config/batchrouting.go]
features: [orchestrator.api-files, orchestrator.api-batches, orchestrator.api-batch-requests]
---

Work that doesn't need an answer now can run cheaper and out of the way of interactive traffic. The orchestrator offers three ways: OpenAI's Batch API over uploaded files, single requests marked as batch requests, and provider service tiers. Files and batches belong to the client key that made them; another key gets `404`.

## Files {#files}

```bash
curl -s http://127.0.0.1:8787/v1/files -H "Authorization: Bearer $VENOM_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl
```

| Endpoint | What it does |
|---|---|
| `POST /v1/files` | Upload (`multipart/form-data`: `file`, `purpose` one of `batch`, `user_data`, `assistants`, `vision`, `evals`, `fine-tune`), at most 200 MB, stored under `$VENOM_DATA_DIR/files/`. Returns the file object (`id: "file-…"`, `bytes`, `filename`, `purpose`, `status: "processed"`). |
| `GET /v1/files?purpose=&limit=&after=` | List, newest first. |
| `GET /v1/files/{id}` | One file object. |
| `GET /v1/files/{id}/content` | The bytes. Generated images are served here too, through a signed link that works without a key for an hour. |
| `DELETE /v1/files/{id}` | Delete; `409 file_in_use` while an unfinished batch reads it. |

Batch results are files of purpose `batch_output`.

## Batches {#batches}

Each line of the input file is `{"custom_id", "method": "POST", "url": <endpoint>, "body"}`:

```json
{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "venom-fast", "messages": [{"role": "user", "content": "Summarise: ..."}]}}
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="my-local-key")
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h")
print(client.batches.retrieve(batch.id).status)
```

| Endpoint | What it does |
|---|---|
| `POST /v1/batches` | `{input_file_id, endpoint, completion_window: "24h", metadata?}`. `endpoint` is `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` or `/v1/responses`. Returns OpenAI's batch object (`id: "batch_…"`, `status`, `request_counts`, `output_file_id`, `error_file_id`). |
| `GET /v1/batches/{id}`, `GET /v1/batches?limit=&after=` | Read and list. |
| `POST /v1/batches/{id}/cancel` | `cancelling`, then `cancelled`; `409 batch_not_cancellable` once it ended. |

Validation (unique `custom_id`, the batch's endpoint, a `model`, no `stream`, at most 50,000 lines) fails the batch with line-numbered `errors`. Output lines are `{id, custom_id, response: {status_code, request_id, body}, error}`: 2xx in the output file, other statuses in the error file, a line that never got an answer with `response: null` and an `error`. A batch past its 24-hour window ends `expired` with what finished.

How it runs: one batch at a time, oldest first, each line served in-process as the batch's key through the normal routing, budgets and request log, on the background lane (flex on OpenAI), with the batch id as its session so the Sessions view sums its cost. Up to 4 lines run at once while no interactive request is in flight, 1 otherwise; while idle, a line for an alias whose first target is in the cloud goes to its first local target. A `429` or `5xx` is retried up to 5 times. State is in SQLite, so a restart resumes a batch where it stopped.

On an upstream with `native_batch: true` (kinds `openai`, `dashscope`, `anthropic`), a batch whose every line resolves there is handed to the provider's own Batch API instead (polled every 30 s, cancel forwarded, results copied back), and each line is logged at half the list price. Anthropic's Message Batches take chat completions only.

## Batch requests {#batch-requests}

Add `x-venom-urgency: batch` (or `x-venom-service-tier: batch`) to a single `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` or `/v1/responses` request and it becomes a batch request. When the model's first target has `native_batch`, it is collected with the key's other requests for that model (for `batch_routing.window`, 30 s by default, or until `max_lines`, 1000) into one provider batch at the batch price. Anything else (and any request under a route profile, or from a key with a hard spend limit) runs at once on the background lane.

```bash
curl -s -i http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
  -H "x-venom-urgency: batch" -H "x-venom-batch-wait: 120" -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"Classify: ..."}]}'
```

The answer is `202` with a `batch_request` object (`id: "breq_…"`, `status` `queued`, `in_progress`, `completed`, `failed`, `cancelled` or `expired`, `mode` `provider_batch` or `background`, and `response` once done), `Location: /v1/batch_requests/{id}` and `x-venom-batch-request`.

| Header or endpoint | What it does |
|---|---|
| `x-venom-batch-wait: <seconds>` | Wait up to that long (at most `batch_routing.max_wait`, 10 min) and answer like the synchronous call, or `202` if it isn't back. |
| `stream: true` (chat completions) | `200 text/event-stream` at once, SSE comments every 15 s, then the result as chunks and `[DONE]`. |
| `x-venom-callback-url` | POST the finished object to that URL; its host must be in `batch_routing.callback_hosts` (else `400 callback_host_not_allowed`). |
| `x-venom-urgency: background` | Serve now on the background lane; `now` changes nothing; any other value is `400`. |
| `GET /v1/batch_requests?limit=&after=` | List, newest first, without responses. |
| `GET /v1/batch_requests/{id}?wait=<seconds>` | Read one; long-poll up to `max_wait`. |
| `POST /v1/batch_requests/{id}/cancel` | Cancel while `queued`; `409 batch_request_not_cancellable` after. |

Batch requests are kept 7 days after they finish.

## Service tiers {#service-tiers}

`x-venom-service-tier: auto | default | flex | priority` asks for a provider tier when the body has no `service_tier` (it wins over a route profile's default). OpenAI gets it as is (flex only on models that offer it; a flex `429` is resent at once on the default tier). Anthropic: `priority` and `auto` become `auto`, `default` becomes `standard_only`. Other kinds ignore it. Any other value is `400 invalid_service_tier`. The answer carries `x-venom-service-tier` with the tier sent. Flex and priority requests are priced at the table's `flex` and `priority` rates, and `GET /admin/v1/spend` reports per tier what was spent and saved against the standard price. Gears can set a tier per role ([Profiles, side requests and gears](/docs/orchestrator/profiles-and-gears.md#gears)).

## Related

- [Configuration reference](/docs/orchestrator/configuration.md#batch-routing) for `batch_routing:`, [Upstreams and kinds](/docs/orchestrator/upstreams.md#keys) for `native_batch`.
