Venom next to pi on the Harness Leaderboard setup
Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.
The Harness Leaderboard runs coding-agent harnesses on the same models and the same tasks through Harbor, and compares them on paired tasks. bun run eval:harbor reproduces one cell of that setup for venom: DeepSWE v1.1 tasks, one model at one effort, with venom and stock pi (configured the way the board configures it) on the same tasks.
It is a maintainer tool in the source checkout (evals/), and it calls a model, so it is opt-in, capped and estimated first like the rest of the eval suite.
Before you start
- A venom source checkout with
make install-evalsdone. - Docker. A native amd64 Linux host matches the board. Docker Desktop on a Mac works with the deviations listed below.
- A running orchestrator that serves the model, and
VENOM_ADMIN_TOKEN. Every trial gets its own key from the orchestrator, limited to the run's model and to what the run may still spend, and the cost is read from the orchestrator's request log. Without the token, a real run refuses to start. - A venom release archive for linux x64, for example from
make release VERSION=… RELEASE_TARGETS=linux/amd64 CLI_TARGETS=linux/x64. - uv, unless you pass your own Harbor 0.24.0 with
--harbor.
Run it
Dry run first. It prints the tasks, the trials, the estimate per trial and the total against the cap, and starts nothing:
cd evals
bun run eval:harbor -- --model zai/glm-5.3-flash --effort low --subset smoke --dry-runThen the run:
bun run eval:harbor -- --model zai/glm-5.3-flash --effort low \
--tasks tengo-destructuring-bindings,httpx-streaming-json-iteration --max-usd 2.5 \
--venom-archive ../dist/release/<v>/venom-<v>-linux-x64.tar.gz| Flag | Default | What |
|---|---|---|
--model |
required | the orchestrator model id both harnesses use |
--effort |
low |
the reasoning effort for both (low, medium, high, …) |
--subset / --tasks / --limit |
smoke |
the board's smoke, pilot or full subset, or named DeepSWE tasks, cut to the first n |
--harnesses |
venom,pi |
either or both |
--k |
1 |
attempts per task and harness |
--max-usd |
VENOM_EVAL_MAX_USD, else 5 |
the hard cap; trials that would cross it are skipped and recorded as skipped |
--concurrency |
2 |
trials at once |
--venom-archive, --venom-sha256 |
the release venom runs from, and its expected hash | |
--base-url, --container-base-url |
VENOM_BASE_URL; the same with host.docker.internal |
the orchestrator for the runner, and as the containers reach it |
--context-window, --max-output-tokens, --input-modalities |
1000000, 131072, text,image |
pi's model entry, as the board's HFPi agent writes it |
--dry-run |
print the plan and the estimate, start nothing |
VENOM_HARBOR points at a Harbor binary and VENOM_HARBOR_CACHE moves the cache (default evals/.cache/harbor, which holds Harbor's venv, the task tarball and each run's job dirs).
What it compares
- pi: pi-coding-agent 1.2.0 installed with
npm cifrom the board's own lockfile, with the model entry the board's HFPi agent writes. - venom: the release archive you give it, run headless (
venom --print --mode json), with every model tier pinned to the run's model and the effort sent asreasoning_effort. - Cost: measured at the orchestrator, outside both harnesses, at list prices.
- Scoring: the board's rules. A trial passes on reward 1. It is
infra, not a failure, when the agent never got to run or the provider failed. The result has the resolve rate per harness with a 95% interval, the cost per trial and per solved task, and a paired gap.
Results go to evals/results/harbor/<run id>.json and .md. A run with fewer than 30 paired tasks says in the result that it is a smoke run, not a measurement. The board's own cells use 111 tasks with 3 attempts each and still report ties within five points.
Deviations on Docker Desktop
Each run records its deviations from the board's setup in the result:
- The tasks ask for no network. Docker Desktop's kernel can't enforce that the way Harbor does it, so the runner gives them a public network.
- Files go into the containers as a tar stream, because
docker cpbreaks the image's file system on Docker Desktop's containerd store. - The task images are amd64. On an Apple Silicon Mac they run emulated, so times are not comparable with the board's. Tasks whose verifier needs AVX (MongoDB in
eicrud-keyset-pagination-cursor) can't be graded there.
The first smoke run
The first run (2026-11-10, evals/results/harbor/ven289-smoke.md) put 8 DeepSWE tasks through both harnesses once each, on zai/glm-5.3-flash at low effort, for $1.37. Docker Desktop stopped partway through, so only two tasks were graded for both. That is enough to show the setup works end to end. It is not enough to say which harness is better.
It did surface one thing about venom: --thinking alone sends no reasoning effort for a model venom has no thinking settings for. The runner sets VENOM_THINKING=<model>=<effort> so venom gets the same effort as pi.
A full DeepSWE comparison (111 tasks, 3 attempts each, both harnesses) costs about $85 at low effort and needs a native amd64 Linux host.