Running evals
Run the eval suite with make eval against an orchestrator or any OpenAI-compatible server, pick tasks with --filter, read pass@1 and pass^k, and keep spend under --max-usd with a dry run first.
make eval runs the eval suite against VENOM_BASE_URL and writes a result file per run. It builds the harness and installs the suite's dependencies first, so from a fresh checkout it is one command.
A first run
make eval-tools # once: the pinned language servers and formatters
make eval ARGS="--k 1 --filter safety --dry-run" # the plan and cost estimate, nothing runs
make eval ARGS="--k 1 --filter safety" # 6 tasks, one trial eachThe run prints one line per trial (PASS, FAIL or SKIP), then a summary, and writes evals/results/<run id>.json and a markdown report next to it. A run id is the start time to the second plus a random 4-hex suffix, such as 2026-10-09T04-15-42Z-3fa1; a name already taken gets -2, -3, and so on, so runs never overwrite each other.
For a decision, run every core task three times:
make eval ARGS="--k 3 --require-tools"The endpoint
The runner reads the same environment as the CLI.
| Variable | What it does | Default |
|---|---|---|
VENOM_BASE_URL |
The OpenAI-compatible endpoint trials call. | http://localhost:8787/v1 |
VENOM_API_KEY |
The key sent to it. | none |
VENOM_MODEL |
The default --model. |
venom-default |
VENOM_FAST_MODEL, VENOM_SMART_MODEL |
The trials' fast and smart model tiers. | the harness defaults |
VENOM_ALIAS_MAP |
Maps venom aliases to real models when you call a provider directly, without the orchestrator: venom-default=gpt-5.4-mini,venom-fast=gpt-5.4-nano. |
none |
VENOM_ORCH_URL |
Points the runner's admin calls at another orchestrator root than the one VENOM_BASE_URL implies. |
derived |
VENOM_ADMIN_TOKEN |
With a reachable orchestrator, each trial gets its own short-lived key, so cost, tokens, cached tokens and time to first token are exact per trial, subagents included. The key is deleted after the trial. | none |
VENOM_EVAL_PROFILE |
The orchestrator route profile trials name (see Spend). | eval when the orchestrator lists it |
VENOM_EVAL_MAX_USD |
The default spend cap. | 5 |
Without an orchestrator, tokens come from the session's own usage and cost is priced from the alias map's target at list price.
Options
Pass them through ARGS: make eval ARGS="--k 3 --filter hard --concurrency 2".
| Option | What it does | Default |
|---|---|---|
--k <n> |
Trials per task. Use 3 for decisions; 1 is a smoke run. | 1 |
--model <id> |
The model or alias trials use. | VENOM_MODEL, else venom-default |
--filter <terms> |
Comma-separated terms: a category, a language, a tag or part of a task id. hard, security and the other opt-in slices run only when named; all runs everything. |
the 43 core tasks |
--concurrency <n> |
Trials at once. | 4 |
--timeout <s> |
Wall-clock limit per trial; the session is aborted at the deadline. | 300 |
--out <dir> |
Where result files go. | evals/results |
--max-usd <usd> |
The spend cap; 0 or inf removes it. |
VENOM_EVAL_MAX_USD, else 5 |
--dry-run |
Print the plan, the per-task estimate and the cap, then exit. | off |
--profile <name> |
The orchestrator route profile; "" for none. A named profile the server lacks exits 2. |
VENOM_EVAL_PROFILE, else eval when listed |
--require-tools |
Exit 2 before any trial unless every pinned server and formatter is found at its pinned version. Use it for any run you will compare or commit. | off |
--thinking <level> |
off, minimal, low, medium, high or xhigh. |
per-model default |
--mode <mode> |
The permission mode trials run in: accept-edits, default or auto. |
accept-edits |
--edit-tool <tool> |
auto, edit, apply_patch or both. |
auto |
--context-file <kind> |
existing (the fixture's own memory files), none, or generated (venom's project analysis written first). |
existing |
--tasks <dir> |
Another task directory, such as tasks mined from your repository. | evals/tasks |
--keep |
Keep each trial's temp workspace and its session transcript. | off |
--list |
List the selected tasks and exit. | off |
--include-quarantined |
Also run tasks quarantined as flaky. | off |
--no-lsp, --no-format |
Turn off LSP diagnostics or the formatter after edits (ablations). | on |
--no-overthink |
Turn off the analysis-paralysis nudges (an ablation). | on |
--arm <arm> |
Which harness the trials run: default (venom as it ships), lean (venom's permission engine, compat and edit tools, without the extra tools and prompt sections) or stock-pi (pi's own prompt and its four tools, no permission engine: leave out the safety and security slices). For ablations; compare arms on the same tasks, model and k. node src/arm-prompt.ts prints each arm's fixed prompt. |
default |
--system-append <file> |
Appends the file's text to every trial's system prompt, after venom's static sections. For measuring a candidate prompt block, such as a per-model note, against a run without it. | off |
How a trial runs
- The fixture is copied to a fresh temp directory and its setup steps run.
- A harness session starts there with an empty temp home (nothing from your own
~/.claudeor~/.venom), an untrusted project and accept-edits mode. Project memory such asAGENTS.mdis still loaded. - Approvals go to a strict headless policy: the project's own test, build and vet commands are allowed, nothing else. No installs, network or git writes. It mirrors a CI run with nobody present.
- One prompt runs with the timeout.
- Hidden tests are overlaid, the checks run, and the trial records pass or fail, tokens, cached tokens, turns, tool calls, denials, wall time and cost.
Reading the results
| Metric | Meaning |
|---|---|
| pass@1 | The per-task success rate, averaged over tasks, with its standard error. |
| pass^k | The chance that all k trials of a task pass, averaged over tasks. An agent gets one try per request, so reliability matters more than pass@k. |
| pass@k | The chance that at least one of k trials passes (also recorded). |
| cost per solved task | Total cost divided by passing trials. |
The markdown report also lists, per trial, where venom stopped something: guard firings, permission denials by reason, sandbox refusals and edit-tool failures, so a regression can be put on the harness or ruled out without reading transcripts.
Spend
Evals are the most expensive thing to run by accident: a full k=3 core run costs about $1.20 to $2.30 on a small model. Every run is capped.
- Estimate first. Before the first trial the runner prints an estimate: per task, the mean cost per trial in the newest earlier result with the same model, thinking level and context-file setting, times k.
--dry-runprints it with the plan and the cap and exits. Check it before any large run:make eval ARGS="--k 3 --filter hard --dry-run". - The cap in the runner. A trial starts only while the finished trials' cost, plus the expected cost of running ones, plus this one's stays within
--max-usd. Trials past the cap print asSKIPand are recorded as skipped, with a budget line in the report. Repetitions are cut before tasks. A capped run is partial: compare it only on what it ran. - The cap at the orchestrator. With
VENOM_ADMIN_TOKEN, each trial's key carries a hard daily budget of what the run may still spend (at least $0.10), so a runaway trial is refused with429 budget_exceededbefore the runner sees its cost. - The flex tier. Trials name the orchestrator's
evalroute profile when it is listed, which asks OpenAI models forservice_tier: flex: half price, slower. With orchestrator accounting, a trial's cost is what the orchestrator priced each request at, so a flex request counts at the flex rate. Compare runs made on the same profile.
Flaky tasks
A task whose checker gives different verdicts on the same workspace is quarantined in evals/quarantine.json. The runner leaves quarantined tasks out and lists them; --include-quarantined runs them anyway. The detector re-runs each task's checks on the untouched fixture and on the reference solution, with no model and at no cost:
cd evals && node src/flake-cli.ts --k 5 --filter allA nightly local lane
node src/nightly-cli.ts (in evals/) runs eight quick core tasks at k=3 against a model on this machine (Ollama's http://127.0.0.1:11434/v1 by default, qwen3:8b and qwen3:4b) and compares each night with the last. A URL that is not loopback is refused, so the lane stays at $0 and offline. Results and history.jsonl go to ~/.venom/evals/nightly/. Schedule it from venom serve with a job on the local gear; see Scheduled agents.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
Exit 2 before any trial with --require-tools |
A pinned language server or formatter is missing or at another version. Run make eval-tools. |
| Exit 2: profile not found | --profile named a route profile the orchestrator doesn't have. Pass --profile "" or add the profile. |
Many SKIP lines |
The cap was reached. Raise --max-usd, or run fewer tasks. |
| Costs show as 0 or estimated | No orchestrator accounting. Set VENOM_ADMIN_TOKEN and point VENOM_BASE_URL at the orchestrator. |
A large task is listed as not judged |
It needs a local clone under $VENOM_EVAL_REPOS. |
Related
- Comparing runs: the paired gate and the committed baseline.
- Eval tools and probes:
make eval-tools. - Gears and route profiles.