Build on Venom Experiments
Define metrics, propose, start, ramp, read and decide A/B tests and guarded rollouts on Venom Flags flags, run product analytics queries and send events, over the Experiments REST API, the event endpoint or MCP.
Venom Experiments runs A/B tests and guarded rollouts on Venom Flags flags. The flag's rollout assigns
units, so there is no second assignment system. Starting, ramping, stopping and deciding are Flags changes made as the
caller: they appear in the flag's history, and flags_flag_revert undoes them. Everything the app does is a REST
operation under /experiments/api/v1 and an MCP tool. Your apps send events and exposures to
/experiments/sdk/events with a Flags environment key.
What you can build
- Experiments from your planning tool. Propose a draft on a flag (
POST /experiments/api/v1/experiments) with a hypothesis, a primary metric and guardrails; the answer has the planned sample and duration. - An agent that watches results. Connect
venom-experimentsand have an agent callexp_experiment_resultseach day andexp_experiment_decidewhen the recommendation is clear. - Guarded rollouts in a release pipeline. Propose with
kind: "rollout"andsteps, start it, and let the sweep ramp the flag while the guardrails hold; pause or resume it withPOST /experiments/api/v1/experiments/{id}/pauseand/resume. - Product analytics in your own dashboard. Run a trend, funnel or retention query with
POST /experiments/api/v1/query(orexp_query_run), or save it as an insight (exp_insight_save) that a Notes page embeds. - A decision log. Subscribe an org webhook to
experiments.decidedand copy each decision, with its note and the flag's audit id, into your own records. - Event tracking from a backend. Send up to 1000 events and 1000 exposures a request to
POST /experiments/sdk/eventswith the environment's server SDK key.
At a glance
Generated from the code on every docs build.
| MCP endpoint | https://cloud.usevenom.dev/workspace/mcp/exp, server name venom-experiments |
| Connect | claude mcp add --transport http venom-experiments https://cloud.usevenom.dev/workspace/mcp/exp |
| MCP tools | 25 (12 read, 13 write) |
| OAuth scopes | experiments:read experiments:write |
| REST twin | none yet |
| Experiments API | 22 operations |
| Webhook events | experiments.guardrail_breached, experiments.decided, experiments.started, experiments.ramped, experiments.stopped |
The older endpoint /experiments/mcp still serves the same tools. /exp/agents in the app shows the connect
commands with your origin filled in.
Authentication and scopes
An experiment is a way of changing a flag, so Experiments checks Flags' permissions: flags:read to read and
flags:write to change. They come from the same places as in Flags:
- A personal API key acts with its member's role: owners, admins and developers read and write; billing and viewer members read.
- A team key reads and writes.
- An OAuth app needs
experiments:read(which stands forflags:read) andexperiments:write(which stands forflags:write), within what the member's role allows. A token for/workspace/mcpor/workspace/mcp/expworks on those endpoints; only a token for theexperimentsresource (/experiments/mcp) works on/experiments/api/v1. - The event endpoint
/experiments/sdk/eventstakes an environment's server SDK key (vfs_…), mobile key (vfm_…) or client-side id (vfc_…), the same keys as Flags' SDK endpoints. - The org needs the Experiments add-on; otherwise each REST call answers
403 addon_requiredand each tool says where to add it. - Rate limits per credential: the REST API shares Flags' 600 requests a minute;
/experiments/mcpallows 240 a minute. Past the limit you get429withRetry-After.
In an environment that requires approval, a start, ramp, stop or decide becomes a Flags approval request instead of
applying, and the answer says so (flagChange.pendingApproval).
Worked example
Define a conversion metric, propose an experiment on the new-checkout flag, start it, then read its results:
BASE="$VENOM_CLOUD/experiments/api/v1"
H="Authorization: Bearer $VENOM_API_KEY"
curl -X POST "$BASE/metrics" -H "$H" -H 'Content-Type: application/json' \
-d '{"environment": "production", "key": "purchase", "type": "conversion", "event": "checkout_completed"}'
curl -X POST "$BASE/experiments" -H "$H" -H 'Content-Type: application/json' -H 'Idempotency-Key: exp-checkout-1' \
-d '{"flag": "new-checkout", "primary": "purchase", "hypothesis": "A one-page checkout converts better.",
"guardrails": [{"metric": "checkout_errors", "tolerated": 0.02}]}'
curl -X POST "$BASE/experiments/<experiment-id>/start" -H "$H" -H 'Content-Type: application/json' -H 'If-Match: "1"' \
-d '{"turnOn": true, "comment": "50/50 in production"}'
curl "$BASE/experiments/<experiment-id>/results" -H "$H"The guardrail metric must exist too. Start, ramp, pause, resume, stop and decide answer
{experiment, flagChange: {auditId, diff, pendingApproval, dryRun}, warnings}; flagChange.auditId is the Flags
history entry to revert. Each answer's ETag is the experiment's version.
The same proposal over MCP, from any client connected to /workspace/mcp/exp:
{ "jsonrpc": "2.0", "id": 1, "method": "tools/call",
"params": { "name": "exp_experiment_propose",
"arguments": { "flag": "new-checkout", "primary": "purchase",
"hypothesis": "A one-page checkout converts better.",
"guardrails": [{ "metric": "checkout_errors", "tolerated": 0.02 }],
"idempotencyKey": "exp-checkout-1" } } }Then exp_experiment_start with {"id": "<experiment-id>", "turnOn": true, "expectVersion": 1} and
exp_experiment_results with {"id": "<experiment-id>"}.
Running an experiment
A typical run for an agent:
exp_metric_list, thenexp_metric_createfor what is missing (count,sum,conversion,ratioorretentionover tracked events;direction: "down"when less is better).exp_experiment_proposewith the flag, the primary metric, a hypothesis and guardrails.exp_plan_get(orGET /experiments/api/v1/plan?primary=…) estimates the sample alone, saving nothing.exp_experiment_start. It is refused when the flag is off in the environment unlessturnOnis true.exp_experiment_resultseach day.recommendation.actionisship,rollback,keep_running,inconclusiveorinvestigate.investigatemeans a sample ratio mismatch: do not act on the numbers.exp_experiment_decidewithdecision(shipwith avariationindex,rollbackoriterate) and areason. It applies the Flags change, writes a decision note into Notes and sendsexperiments.decided.
Writes take expectVersion (or If-Match), idempotencyKey (or Idempotency-Key; a retry within 24 hours returns
the first answer), comment (shown in the flag's history) and dryRun. Shares are in thousandths of a percent
(50000 = 50 %), between 1000 and 99000. Ramping restarts the analysis, so ramp early and rarely. A metric an experiment
uses can only be renamed.
A guarded rollout (kind: "rollout") needs at least one guardrail. Its default ladder is 1 %, 5 %, 25 %, 50 %, then
100 %, a day each. The sweep advances it, promotes it after the last step, or rolls it back and files a Tasks issue
when a guardrail breaks. exp_experiment_rollout (action: "pause" or "resume") holds it; a breach still rolls a
paused rollout back.
Sending events
POST /experiments/sdk/events takes {"events": [...], "exposures": [...]}: up to 1000 of each, 512 KB. An event is
{event, unit: {kind, key}, value?, props?, ts?}; props is at most 2 KB of JSON. Invalid entries are dropped and
counted in the 202 answer. Past the org's daily cap it answers 429 with Retry-After. See
Sending events for the TypeScript SDK's track() and retention.
Events
5 org webhook events, generated from lib/webhooks/events.ts. Subscribe an endpoint to them in Settings, Webhooks (see Webhooks and events).
| Event | When |
|---|---|
experiments.guardrail_breached |
A guardrail metric of a running experiment is worse than its tolerated regression (once per experiment and metric). |
experiments.decided |
An experiment was decided: a variation shipped, a rollback or an iteration, by a person, an agent or the automatic guardrail rollback. |
experiments.started |
An experiment started serving its treatments (by a person, an agent, or an approved request): the experiment, flag, environment, the treatments' share and who started it. |
experiments.ramped |
A running experiment's treatment share changed (by hand, or a guarded rollout's next step): the share before and after, and who. |
experiments.stopped |
A running experiment was stopped and control is served to everyone: the experiment, flag, environment and who stopped it. |
The data of both events has the experiment's experiment id, name, flag, project, environment and url.
experiments.decided adds decision, variation, reason, note (the decision note's path) and auditId (the flag's
history entry); an automatic rollback adds automatic: true. experiments.guardrail_breached adds metric, arm,
harm, tolerated, p and autoRollback, and is sent once per experiment and metric.
Experiments has no webhooks of its own. The flag changes it makes also reach Flags webhooks as
flag.update.
REST routes
The Experiments API: 22 operations, generated from services/nodejs-cloud/contracts/experiments-openapi.yaml. "Key or token" is a Venom API key or an OAuth access token; SDK keys and client ids belong to an environment and are for your app's runtime.
/experiments
| Method | Path | Summary | Auth |
|---|---|---|---|
| POST | /experiments/sdk/events |
Send events and exposures | SDK key or client id |
| GET | /experiments/api/v1/experiments |
List experiments | key or token |
| POST | /experiments/api/v1/experiments |
Propose an experiment (a draft; nothing is served yet) | key or token |
| GET | /experiments/api/v1/experiments/{id} |
Get an experiment | key or token |
| PATCH | /experiments/api/v1/experiments/{id} |
Edit a draft | key or token |
| GET | /experiments/api/v1/experiments/{id}/results |
Results from the daily rollups, cached until they change | key or token |
| GET | /experiments/api/v1/experiments/{id}/timeline |
The primary metric's cumulative lift by day, for a chart | key or token |
| POST | /experiments/api/v1/experiments/{id}/{action} |
Start, ramp, stop or decide an experiment (each is a Flags change as the caller) | key or token |
/events
| Method | Path | Summary | Auth |
|---|---|---|---|
| GET | /experiments/api/v1/events |
The events explorer - event names with 14 days of volume, and a sample of recent events | key or token |
/plan
| Method | Path | Summary | Auth |
|---|---|---|---|
| GET | /experiments/api/v1/plan |
Plan an experiment - the sample it needs and how long recent traffic takes to deliver it (nothing is saved) | key or token |
/query
| Method | Path | Summary | Auth |
|---|---|---|---|
| POST | /experiments/api/v1/query |
Run a trend, funnel or retention query (read-only, time-limited; nothing is saved) | key or token |
/insights
| Method | Path | Summary | Auth |
|---|---|---|---|
| GET | /experiments/api/v1/insights |
List saved insights of an environment | key or token |
| POST | /experiments/api/v1/insights |
Save a query as an insight | key or token |
| GET | /experiments/api/v1/insights/{id} |
One insight | key or token |
| PATCH | /experiments/api/v1/insights/{id} |
Rename an insight or change its query (If-Match with its version) | key or token |
| DELETE | /experiments/api/v1/insights/{id} |
Delete an insight (Notes pages that embed it show "Insight unavailable") | key or token |
| GET | /experiments/api/v1/insights/{id}/result |
Run a saved insight now - what a Notes page embedding it shows | key or token |
/metrics
| Method | Path | Summary | Auth |
|---|---|---|---|
| GET | /experiments/api/v1/metrics |
List metrics of an environment | key or token |
| POST | /experiments/api/v1/metrics |
Create a metric | key or token |
| GET | /experiments/api/v1/metrics/{key} |
Get a metric | key or token |
| PATCH | /experiments/api/v1/metrics/{key} |
Update a metric (not what it measures while an experiment uses it) | key or token |
| DELETE | /experiments/api/v1/metrics/{key} |
Delete a metric no experiment uses | key or token |
MCP tools
25 tools: 12 read, 13 write. Generated from the code; "write (destructive)" is the tool's destructiveHint. A tool with a REST route is the same call over HTTP.
| Tool | Title | Access | Scope | REST route |
|---|---|---|---|---|
exp_event_list |
List events | read | experiments:read |
|
exp_experiment_decide |
Decide an experiment | write | experiments:write |
|
exp_experiment_get |
Get an experiment | read | experiments:read |
|
exp_experiment_list |
List experiments | read | experiments:read |
|
exp_experiment_propose |
Propose an experiment | write | experiments:write |
|
exp_experiment_ramp |
Ramp an experiment | write | experiments:write |
|
exp_experiment_results |
Experiment results | read | experiments:read |
|
exp_experiment_rollout |
Pause or resume a rollout | write | experiments:write |
|
exp_experiment_start |
Start an experiment | write | experiments:write |
|
exp_experiment_stop |
Stop an experiment | write | experiments:write |
|
exp_experiment_timeline |
Experiment lift over time | read | experiments:read |
|
exp_experiment_update |
Change a draft experiment | write | experiments:write |
|
exp_insight_delete |
Delete an insight | write (destructive) | experiments:write |
|
exp_insight_get |
Read an insight | read | experiments:read |
|
exp_insight_list |
List insights | read | experiments:read |
|
exp_insight_result |
Run an insight | read | experiments:read |
|
exp_insight_save |
Save an insight | write | experiments:write |
|
exp_insight_update |
Change an insight | write | experiments:write |
|
exp_metric_create |
Create a metric | write | experiments:write |
|
exp_metric_delete |
Delete a metric | write (destructive) | experiments:write |
|
exp_metric_get |
Read a metric | read | experiments:read |
|
exp_metric_list |
List metrics | read | experiments:read |
|
exp_metric_update |
Change a metric | write | experiments:write |
|
exp_plan_get |
Plan a sample size | read | experiments:read |
|
exp_query_run |
Run a trend, funnel or retention query | read | experiments:read |
Each tool takes format: "json" for a structured answer instead of compact markdown. project defaults to the org's
default Flags project and environment to production. The tool list is the same for every caller; a read-only
credential is refused in words when it calls a write tool.
Related
- Venom Experiments: metrics, the analysis, sample ratio mismatch and guarded rollouts.
- Sending events and Insights and dashboards.
- Build on Venom Flags: the flags experiments run on, and how to revert a change.
- Authentication and scopes, Errors, limits and pagination, Webhooks and events and MCP.