Eval tools and probes
Pin the language servers and formatters every eval trial uses with make eval-tools, check which reasoning efforts each provider really accepts with make effort-probe, and list MCP Registry servers for the cloud catalog with make mcp-catalog-sync.
Three maintainer tools sit next to the eval suite. make eval-tools makes trials reproducible. make effort-probe and make mcp-catalog-sync check venom's tables against the outside world on demand; neither runs in tests.
make eval-tools
Trials run with an empty temp home, so which language servers and formatters they find must not depend on what happens to be installed on the machine. The suite pins its own in evals/tools.json:
| Tool | Pin |
|---|---|
| typescript-language-server | 6.0.1, with typescript 5.9.3 next to it |
| prettier | 3.9.9 |
| gopls | v0.23.0 (go install) |
| gofmt | the Go toolchain's (required, not pinned) |
make eval-tools # install into evals/.tools/
make eval-tools ARGS=--force # reinstallIt installs exact versions into evals/.tools/ (gitignored), nothing global, and links the binaries into evals/.tools/bin. It does nothing when .tools/installed.json already matches the manifest. It ends by printing what a trial finds and exits 1 if that misses the manifest. It is the one eval step that needs the network; runs don't.
The runner puts evals/.tools/bin first on PATH and in VENOM_TOOL_DIRS, which the harness searches before PATH for language servers and formatters, so the pinned binaries win. Every result records each tool found with its path and version, and make eval-compare warns when two runs differ. Run with --require-tools to refuse a run whose tools don't match (Running evals).
make effort-probe
Providers accept different reasoning_effort levels, and the lists change. The probe sends one tiny chat completion (16 output tokens) per level of the ladder none, minimal, low, medium, high, xhigh, max, plus a bogus level, to each provider whose key is in the environment, and compares what worked with what venom declares: the cloud's effort adapter and the orchestrator's per-model quirks.
make effort-probe ARGS="--dry-run" # the plan and estimate
make effort-probe ARGS="--provider openai,qwen --max-usd 0.25"
make effort-probe ARGS="--orchestrator http://127.0.0.1:8787 --paths go" # through a running orchestratorEach probed model gets up to three rows:
| Path | What is sent | Must |
|---|---|---|
raw |
The level as is, straight to the provider. | Shows which levels the provider rejects (400). |
adapter |
The body after the cloud's effort adapter rewrote it. | Never 400. |
go |
The level as is, through a running orchestrator (model upstream/model), so its quirks apply. |
Never 400. |
A model that also accepts the bogus level ignores the field, so its raw row proves nothing.
| Option | What it does | Default |
|---|---|---|
--max-usd <usd> |
Refuses to start when the upper-bound estimate exceeds it, then stops at the measured spend. | 1 |
--dry-run |
Print the plan and estimate, send nothing. | off |
--provider <a,b> |
Only these providers. | every keyed one |
--models <p=m1,m2> |
Override a provider's model list. | built-in list |
--orchestrator <url> |
The orchestrator for the go path. |
none (no go rows) |
--paths <list> |
raw, adapter, go. |
all three |
--env-file <file> |
Where to read keys missing from the environment. Values are never printed. | the checkout's .env |
--out <file> |
Where the JSON report goes. | dist/effort-probe/<time>.json |
Exit 0 when everything matches, 1 on any mismatch, 2 on a usage or budget error. It is live: it needs keys and the network and costs a few cents. Report the cost of each run.
make mcp-catalog-sync
Venom Cloud keeps a curated catalog of MCP servers an org policy can offer. make mcp-catalog-sync lists candidates for it from the official MCP Registry: active servers a policy could carry without a secret (a streamable-HTTP remote with no required header and no URL variable; with --packages, also npm or PyPI stdio packages with no required environment variable or argument), minus the ones the catalog already has. It prints a table, or JSON. It never edits the catalog; each candidate is checked against the vendor's own docs and added by hand.
make mcp-catalog-sync
make mcp-catalog-sync ARGS="--search docs --packages --json"| Option | What it does | Default |
|---|---|---|
--registry <url> |
The registry to page through. | https://registry.modelcontextprotocol.io |
--input <file> |
Read a saved registry response instead of the network. | none |
--catalog <file> |
The catalog to subtract. | the cloud's catalog.json |
--packages |
Include stdio packages, not only remotes. | off |
--search <text> |
Only servers matching the text. | all |
--max-pages <n> |
Pages of 100 to read. | 1000 |
--json |
JSON instead of a table. | off |
It is maintainer-side and on demand only: the cloud never calls a registry at runtime.
Related
- Running evals.
- Models and reasoning: the effort levels a session asks for.
- Search and MCP: MCP through the orchestrator.