---
title: Venom next to pi on the Harness Leaderboard setup
description: Run venom and stock pi side by side on the Harness Leaderboard's DeepSWE setup through Harbor, one model at one effort, with a spend cap and a dry run first.
section: evals
order: 45
audience: [operator, builder]
sources: [evals/src/harbor.ts, evals/src/harbor-cli.ts, evals/harbor, evals/AGENTS.md]
features: []
---

The [Harness Leaderboard](https://huggingface.co/spaces/joelniklaus/harness-leaderboard) runs coding-agent harnesses on the same models and the same tasks through [Harbor](https://github.com/laude-institute/harbor), and compares them on paired tasks. `bun run eval:harbor` reproduces one cell of that setup for venom: DeepSWE v1.1 tasks, one model at one effort, with venom and stock pi (configured the way the board configures it) on the same tasks.

It is a maintainer tool in the source checkout (`evals/`), and it calls a model, so it is opt-in, capped and estimated first like the rest of the eval suite.

## Before you start {#requirements}

- A venom source checkout with `make install-evals` done.
- Docker. A native amd64 Linux host matches the board. Docker Desktop on a Mac works with the deviations listed below.
- A running orchestrator that serves the model, and `VENOM_ADMIN_TOKEN`. Every trial gets its own key from the orchestrator, limited to the run's model and to what the run may still spend, and the cost is read from the orchestrator's request log. Without the token, a real run refuses to start.
- A venom release archive for linux x64, for example from `make release VERSION=… RELEASE_TARGETS=linux/amd64 CLI_TARGETS=linux/x64`.
- [uv](https://docs.astral.sh/uv/), unless you pass your own Harbor 0.24.0 with `--harbor`.

## Run it {#run}

Dry run first. It prints the tasks, the trials, the estimate per trial and the total against the cap, and starts nothing:

```bash
cd evals
bun run eval:harbor -- --model zai/glm-5.3-flash --effort low --subset smoke --dry-run
```

Then the run:

```bash
bun run eval:harbor -- --model zai/glm-5.3-flash --effort low \
  --tasks tengo-destructuring-bindings,httpx-streaming-json-iteration --max-usd 2.5 \
  --venom-archive ../dist/release/<v>/venom-<v>-linux-x64.tar.gz
```

| Flag | Default | What |
|---|---|---|
| `--model` | required | the orchestrator model id both harnesses use |
| `--effort` | `low` | the reasoning effort for both (`low`, `medium`, `high`, …) |
| `--subset` / `--tasks` / `--limit` | `smoke` | the board's `smoke`, `pilot` or `full` subset, or named DeepSWE tasks, cut to the first n |
| `--harnesses` | `venom,pi` | either or both |
| `--k` | `1` | attempts per task and harness |
| `--max-usd` | `VENOM_EVAL_MAX_USD`, else 5 | the hard cap; trials that would cross it are skipped and recorded as skipped |
| `--concurrency` | `2` | trials at once |
| `--venom-archive`, `--venom-sha256` | | the release venom runs from, and its expected hash |
| `--base-url`, `--container-base-url` | `VENOM_BASE_URL`; the same with `host.docker.internal` | the orchestrator for the runner, and as the containers reach it |
| `--context-window`, `--max-output-tokens`, `--input-modalities` | 1000000, 131072, `text,image` | pi's model entry, as the board's HFPi agent writes it |
| `--dry-run` | | print the plan and the estimate, start nothing |

`VENOM_HARBOR` points at a Harbor binary and `VENOM_HARBOR_CACHE` moves the cache (default `evals/.cache/harbor`, which holds Harbor's venv, the task tarball and each run's job dirs).

## What it compares {#what}

- **pi**: pi-coding-agent 1.2.0 installed with `npm ci` from the board's own lockfile, with the model entry the board's HFPi agent writes.
- **venom**: the release archive you give it, run headless (`venom --print --mode json`), with every model tier pinned to the run's model and the effort sent as `reasoning_effort`.
- **Cost**: measured at the orchestrator, outside both harnesses, at list prices.
- **Scoring**: the board's rules. A trial passes on reward 1. It is `infra`, not a failure, when the agent never got to run or the provider failed. The result has the resolve rate per harness with a 95% interval, the cost per trial and per solved task, and a paired gap.

Results go to `evals/results/harbor/<run id>.json` and `.md`. A run with fewer than 30 paired tasks says in the result that it is a smoke run, not a measurement. The board's own cells use 111 tasks with 3 attempts each and still report ties within five points.

## Deviations on Docker Desktop {#deviations}

Each run records its deviations from the board's setup in the result:

- The tasks ask for no network. Docker Desktop's kernel can't enforce that the way Harbor does it, so the runner gives them a public network.
- Files go into the containers as a tar stream, because `docker cp` breaks the image's file system on Docker Desktop's containerd store.
- The task images are amd64. On an Apple Silicon Mac they run emulated, so times are not comparable with the board's. Tasks whose verifier needs AVX (MongoDB in `eicrud-keyset-pagination-cursor`) can't be graded there.

## The first smoke run {#smoke}

The first run (2026-11-10, `evals/results/harbor/ven289-smoke.md`) put 8 DeepSWE tasks through both harnesses once each, on `zai/glm-5.3-flash` at low effort, for $1.37. Docker Desktop stopped partway through, so only two tasks were graded for both. That is enough to show the setup works end to end. It is not enough to say which harness is better.

It did surface one thing about venom: `--thinking` alone sends no reasoning effort for a model venom has no thinking settings for. The runner sets `VENOM_THINKING=<model>=<effort>` so venom gets the same effort as pi.

A full DeepSWE comparison (111 tasks, 3 attempts each, both harnesses) costs about $85 at low effort and needs a native amd64 Linux host.
