Skip to content
Docs menu

Model quality, scorecard and red team

Schedule quality checks on the models your gears use, catch drift and swap degraded models, read the agent scorecard per gear or key group, and run the red-team suite from the console.

View .md

Three pages answer "are our models and agents still doing well?":

Page What it is for Read Change
/quality scheduled quality checks: budget, flags, scoreboard, suites, auto-swap, run history routing:read (owner, admin, developer, viewer) org:manage (owner, admin)
/quality/suites/{id} the suite editor, with a Test button per case routing:read org:manage
/quality/runs/{id} one run's per-case results: answer, grade, latency routing:read
/scorecard how each gear's sessions went over 7, 30 or 90 days usage:read org:manage for the outcome-report switch
/redteam the red-team suite: score per category, every case, the trend audit:read (owner, admin, developer) org:manage to run

Turn on quality checks

Quality checks spend the org's money on a schedule, so they are off until an owner or admin turns them on with a monthly cap.

  1. Open /quality. The Checks and budget card shows the status, this month's spend, an upper-bound estimate for a month and the runs per month.
  2. Under Suites, add a starter suite (Basic reasoning reasoning, Tool calling tools or JSON output json) and pick the models it runs against.
  3. Set Monthly cap (USD), required and above $0, and turn checks on. "Quality checks are on, capped at $N a month."

A case starts only while the month's checks plus its estimate stay within the cap; a reached org spend cap stops runs too. Checks run with the org's internal key and are labelled, so they count toward spend but are left out of traffic figures.

Suites and cases

Suite field Default Notes
Name, Description
Schedule daily hourly (744 runs a month per model) or daily (31)
Models each picked model runs every case on the schedule; the same ids as the Gears page
Judge model none grades llm_judge cases; its calls count toward the cap
Run on schedule on paused suites keep their runs and scores

Each case has a prompt, an optional system message, tools and max_tokens, and one grader:

Grader Passes when
exact the answer equals the expected text (optionally ignoring case)
contains the answer contains every expected string
regex the answer matches the pattern
json_schema the answer is JSON valid against the schema
number the answer is the expected number, within an optional tolerance
tool_call the model calls the named tool, optionally with the given arguments
llm_judge the judge model says it passes

Test in the editor runs one case against a model once, before you save.

Drift and auto-swap

The Scoreboard lists each model and suite with its score, a sparkline, latency p50, cost per run and status. A model is flagged degraded when:

  • its pass rate over the last 3 runs is significantly lower (one-sided test at about p = 0.01) than over the 20 runs before, and at least 10 points lower, with enough graded cases on both sides. The flag clears when the recent pass rate is back within 5 points of the frozen baseline;
  • or every one of the last 3 runs' median latency is above the baseline's 90th percentile and their median is at least 1.5 times the baseline median. It clears within 1.2 times.

The nav link's badge counts degraded models. Auto-swap (off for every role by default) lets you pick gear roles that list two or more models: for those, a degraded model is left out of the role's list in /v1/gears, so venom uses the next model, until the model recovers. Every swap and swap-back is in the audit log.

Scorecard

/scorecard compares groups of sessions with the org median. Query parameters: ?by=gear (default), ?by=team (by key group), or ?by=member (owners and admins only, and the page never ranks people), and ?days=7, 30 (default) or 90.

Column Meaning
Completed sessions that ended cleanly, of those with a known end
Abandoned sessions closed mid-run or after a failed run
Rework sessions followed by another on the same branch within 24 hours, or that edited files back
Tool errors failed tool calls of all tool calls venom reported
Stuck shifts times the stuck detector shifted up a gear, per reported session
Escalations, Escalations worked work re-run on a stronger tier, and how often the re-run passed
Cost / session, Cost / completed everything the sessions cost, subagents included
First output median time to the first answer or tool call of a run
Quality pass rate of the gear's main models in the quality checks above

Groups with fewer than 5 sessions get no comparison. The data comes from request logs plus session outcome reports: counts venom sends when a session ends (outcome, turns, tool calls and errors, never text). Reports are on by default; the switch on this page turns them off for the org.

Red team

/redteam runs known attacks against your guardrails: prompt injection, jailbreaks, data exfiltration, secret leakage, policy bypass and tool abuse. It shows the latest run's score, a line per category, every case with the guardrail code that caught it (linked to its /g/<code> page) or the catalog entry that would, and the trend with each run's cost.

Run checks now runs the free cases (no model calls) on the org's guardrails, optionally for one key group. Cases that need a model run from the CLI, through your routing and capped:

venom redteam run --max-usd 1

Runs are kept and audited as redteam.run. See Red team for the corpus and the CLI.

Troubleshooting

Message Cause and fix
400 cap_required set a monthly cap above $0 before turning checks on
"Your role cannot change quality checks (org:manage)." ask an owner or admin
"No gear role lists two or more models yet." add a fallback model to a role on the Gears page, then allow auto-swap
The scorecard has no member view grouping by member needs org:manage
Checks stopped mid-month the monthly cap or the org's spend cap was reached