---
title: Cloud eval runner
description: Run the evals your team queues in Venom Cloud on a machine you control with venom eval run --cloud, with a spend cap per run and no provider keys handed to the trials.
section: evals
order: 40
audience: [user, operator]
sources: [cli/src/eval-command.ts, cli/src/cli.ts, evals/src/cloud-runner.ts, evals/AGENTS.md]
features: [evals.eval]
---

Venom Cloud's console can queue eval runs (golden tasks, gear A against gear B, a library prompt or skill under test), but the cloud has no sandbox and runs no trial itself. `venom eval run --cloud` turns a machine you control into the runner that does.

## Start a runner {#example}

```bash
venom cloud login                                        # once: the key the runner uses
cd ~/src/venom                                           # a venom checkout with evals/ installed
venom eval run --cloud --name ci-box --max-usd 10
```

The runner polls the cloud for runs queued to it, so nothing listens on the machine. It says hello, claims a queued run every 5 seconds, runs the eval suite once per variant (gear A, B, and so on), posts progress at least every 15 seconds, and posts the result: per task, each trial's pass, cost, wall time, error and failed checks. The run then shows in the console's Evals page.

## Flags {#flags}

| Flag | What it does | Default |
|---|---|---|
| `--cloud` | Required: run the cloud's queue. | none |
| `--name <runner>` | The runner's name in the console. | this host's name |
| `--max-usd <usd>` | The most one run may spend on this machine. The cloud never queues a bigger run to it, and a bigger one is refused here too. | `VENOM_EVAL_MAX_USD`, else `5` |
| `--repos <dir>` | Where golden tasks' repository names resolve. | the working directory |
| `--concurrency <n>` | Trials at once. | `4` |
| `--poll-ms <ms>` | How often to ask for a queued run (at least 1000). Not in the usage text. | the interval the cloud suggests, else 5000 |
| `--once` | Run at most one queued run, then exit. | off |

`venom eval` with no arguments, `help` or `--help` prints the usage.

## Requirements {#requirements}

- **A Venom Cloud login**: `venom cloud login`, or `VENOM_CLOUD_URL` and `VENOM_CLOUD_KEY` in the environment. They go to the runner in its environment, never on its command line, where any local user could read them.
- **An eval suite**: `VENOM_EVALS_DIR`, else the nearest `evals/` directory above where you run it, installed once with `make install-evals build-harness`.

## Spend and credentials {#spend}

- Each variant runs with `--max-usd` set to what the run has left, split over the variants still to run.
- Trials call the cloud with the run's own key, which the cloud minted with a hard budget of the run's cap and revokes when the run ends. When the cloud answers progress with stop (capped, cancelled or lost), the trials in flight get 60 seconds to end, then are killed.
- The trials' environment is the runner's minus every provider key, token and `VENOM_*` setting, plus the run's own endpoint and key. A model the cloud doesn't list fails with "No API key found" instead of quietly calling a provider with a key from your machine.

## Golden tasks and variants {#golden}

- A golden task's repository is a git URL, cloned bare once into `~/.venom/evals/repos/` and fetched for each run with the machine's own git credentials, or a name under `--repos`. Its ref is resolved to a commit, the fixture is that commit, and its check is a command.
- A variant's gear gives the model (its first concrete main model), the thinking level, the route profile and the fast model.
- A library prompt under test wraps every task prompt the way `/org:<name> <prompt>` would; a library skill under test is installed in every trial's home.

## Errors {#errors}

| Message | Fix |
|---|---|
| `not logged in to Venom Cloud` (exit 2) | Run `venom cloud login`, or set `VENOM_CLOUD_URL` and `VENOM_CLOUD_KEY`. |
| `no eval suite found` (exit 2) | Run it inside a venom checkout, or set `VENOM_EVALS_DIR`. |
| `... is not installed yet` (exit 2) | Run `make install-evals build-harness` in the checkout. |
| `unknown option` or `needs a value` (exit 2) | Check the flags above. |

## Related

- [Running evals](/docs/evals/running.md): the suite each variant runs.
- [Background agents](/docs/automation/background.md): `venom intake run`, the same polling runner pattern for issues.
