Skip to content
Docs menu

Evals and experiments

Run the eval suite or your own golden tasks against your gears on a runner you control, and A/B test two gears or two prompt versions on live sessions from the Venom Cloud console.

View .md

Two ways to tell whether a change made things better:

  • Evals run fixed tasks (venom's built-in eval suite or your team's golden tasks) with each gear, on a runner machine you control, and compare the pass rates. The cloud runs no trial itself.
  • Experiments split real sessions between two gears, or two versions of a library prompt, and compare cost and quality signals.
Page What it is for Read Change
/evals suggestions, the new-run form with a cost estimate, run history, golden tasks, runners routing:read (owner, admin, developer, viewer) org:manage to queue runs (they spend), edit golden tasks and remove runners
/evals/runs/{id} one run: status (refreshed while it runs), figures, verdicts, per-task results (?changed=1 shows only tasks whose result changed) routing:read org:manage to cancel
/evals/golden/new a new golden task; ?share=<token> drafts one from a shared session org:manage
/evals/golden/{id} edit or delete a golden task routing:read org:manage
/experiments the running gear experiment, the start form, past experiments, prompt tests, the default gear routing:read org:manage
/experiments/{id} one gear experiment's results routing:read org:manage
/experiments/prompts/{id} one prompt A/B test's results routing:read org:manage

Compare two gears with an eval run

  1. Start a runner on a machine with the repositories and credentials the tasks need. /evals shows the same commands:

    venom cloud login
    venom eval run --cloud --name ci-box --max-usd 10
  2. On /evals, in the new-run form, pick Gear A and Gear B, the task set (built-in categories such as bugfix and feature, or your golden tasks), Trials per task (k) and a Cap (USD). The form shows a live cost estimate.

  3. Queue it. "Queued for ci-box. It starts at the runner's next poll."

  4. Open the run at /evals/runs/{id} for the verdict per pair of gears.

Evals

The run form

Field Default Notes
Gear A, B, … the org's first two gears each gear's main model is shown under it
Task set built-in (categories bugfix and feature ticked), or golden tasks when there is no suite opt-in categories are marked
Trials per task (k) 3 1 to 10; pass^k needs k ≥ 2
Cap (USD) none the run's key stops at it and the runner stops starting trials
Runner the first runner offline runners pick the run up when they start; a runner may set its own maximum

When a run is claimed the cloud mints a key just for it, with hard daily and monthly budgets equal to the cap, valid for 12 hours. The key is revoked when the run finishes, is cancelled, or is lost (no progress from the runner). Cancel says "Cancelled. The run's key is revoked, so nothing more is billed to it."

Verdicts

Verdict Meaning
Better B passed significantly more tasks than A
Worse B passed significantly fewer
No clear difference the difference is within noise
Nothing to compare not enough finished trials

Golden tasks

A golden task is a real task from your team, with a check that tells whether it was done.

Field Notes
Name how it shows in runs and results
Prompt what the agent is asked, as a person on the team asked it
Repository a git URL the runner clones with its own credentials, or a folder name under the runner's --repos directory. No passwords or tokens in the URL
Ref the commit, tag or branch the task starts from (before the fix)
Setup optional commands run in the fresh checkout before the agent starts, one per line
Check command run after the agent finishes; exit 0 means the task is done
Timeout (seconds) the agent's time for one trial, 30 to 3600

Save as golden task on a session's page drafts one from that session.

Suggestions and library tests

After someone changes a gear on the Gears page, /evals suggests running the golden tasks with the gear before and after; Run queues it and Dismiss hides it. A library prompt's page has a Test card that runs the golden tasks with version A (baseline) and version B (the change) of that prompt. Removing a runner cancels its queued and running runs.

Gear experiments

One gear experiment runs at a time per org. The start form on /experiments:

Field Default Notes
Gear A (control), Gear B (candidate) two of the org's gears; while it runs, the Gears page refuses a change that drops either
On B (%) 50 1 to 99: the share of new sessions on B
Who takes part every client key or the keys of chosen members, or of chosen key groups
Run for (days) none 1 to 90
Or stop at (sessions) none with neither, it runs until stopped
Members may opt out on members opt out in venom with /gear experiment off
Name optional

A new venom session that starts in gear A (or with no gear), without --model, joins. Its arm comes from a hash of the experiment id and session id, so a resumed session keeps its arm. Only counts are recorded, never content.

/experiments/{id} shows per arm: sessions, cost and tokens per session, latency p50 and p95, error, abandon and tool error rates, stuck shifts per session and the scheduled quality checks on each arm's main model. Below 30 finished sessions per arm there is no verdict. The verdict reads like "eco is 23% cheaper, no significant quality difference (n=412 vs 398)".

Control Effect
Stop ends the experiment
Make B the default (promote) that arm's gear becomes the org's default gear and the experiment ends as promoted
Delete removes an ended experiment and its sessions
Default gear the gear new sessions start in unless the user picks one; or none

Prompt A/B tests

From a library prompt's page, Test on live sessions starts a test of two versions:

Field Default Notes
Prompt the page's prompt must have a published version
Version A (control) the published version
Version B (candidate) a version number or the current draft
On B (%), Who takes part, Run for (days), Or stop at (sessions), Name as for gear experiments

Both versions' text is copied when the test starts, so later edits never change it. One test runs per prompt. Promoting an arm publishes its text as the next version (nothing is published when it already is the published version) and ends the test. Results are at /experiments/prompts/{id} and listed on /experiments.

Troubleshooting

Message Cause and fix
"Your role cannot run experiments (org:manage)." owners and admins start, stop and promote experiments
409 experiment_running another gear experiment (or a test of this prompt) is running; stop it first
409 not_published the prompt has no published version yet
400 same_text both versions have the same text and parameters
409 no_draft version B is "draft" but the prompt has none
A queued run never starts its runner is offline; start venom eval run --cloud on the runner machine