Skip to content
Docs menu

Build on Venom Experiments

Define metrics, propose, start, ramp, read and decide A/B tests and guarded rollouts on Venom Flags flags, run product analytics queries and send events, over the Experiments REST API, the event endpoint or MCP.

View .md

Venom Experiments runs A/B tests and guarded rollouts on Venom Flags flags. The flag's rollout assigns units, so there is no second assignment system. Starting, ramping, stopping and deciding are Flags changes made as the caller: they appear in the flag's history, and flags_flag_revert undoes them. Everything the app does is a REST operation under /experiments/api/v1 and an MCP tool. Your apps send events and exposures to /experiments/sdk/events with a Flags environment key.

What you can build

  • Experiments from your planning tool. Propose a draft on a flag (POST /experiments/api/v1/experiments) with a hypothesis, a primary metric and guardrails; the answer has the planned sample and duration.
  • An agent that watches results. Connect venom-experiments and have an agent call exp_experiment_results each day and exp_experiment_decide when the recommendation is clear.
  • Guarded rollouts in a release pipeline. Propose with kind: "rollout" and steps, start it, and let the sweep ramp the flag while the guardrails hold; pause or resume it with POST /experiments/api/v1/experiments/{id}/pause and /resume.
  • Product analytics in your own dashboard. Run a trend, funnel or retention query with POST /experiments/api/v1/query (or exp_query_run), or save it as an insight (exp_insight_save) that a Notes page embeds.
  • A decision log. Subscribe an org webhook to experiments.decided and copy each decision, with its note and the flag's audit id, into your own records.
  • Event tracking from a backend. Send up to 1000 events and 1000 exposures a request to POST /experiments/sdk/events with the environment's server SDK key.

At a glance

Generated from the code on every docs build.

MCP endpoint https://cloud.usevenom.dev/workspace/mcp/exp, server name venom-experiments
Connect claude mcp add --transport http venom-experiments https://cloud.usevenom.dev/workspace/mcp/exp
MCP tools 25 (12 read, 13 write)
OAuth scopes experiments:read experiments:write
REST twin none yet
Experiments API 22 operations
Webhook events experiments.guardrail_breached, experiments.decided, experiments.started, experiments.ramped, experiments.stopped

The older endpoint /experiments/mcp still serves the same tools. /exp/agents in the app shows the connect commands with your origin filled in.

Authentication and scopes

An experiment is a way of changing a flag, so Experiments checks Flags' permissions: flags:read to read and flags:write to change. They come from the same places as in Flags:

  • A personal API key acts with its member's role: owners, admins and developers read and write; billing and viewer members read.
  • A team key reads and writes.
  • An OAuth app needs experiments:read (which stands for flags:read) and experiments:write (which stands for flags:write), within what the member's role allows. A token for /workspace/mcp or /workspace/mcp/exp works on those endpoints; only a token for the experiments resource (/experiments/mcp) works on /experiments/api/v1.
  • The event endpoint /experiments/sdk/events takes an environment's server SDK key (vfs_…), mobile key (vfm_…) or client-side id (vfc_…), the same keys as Flags' SDK endpoints.
  • The org needs the Experiments add-on; otherwise each REST call answers 403 addon_required and each tool says where to add it.
  • Rate limits per credential: the REST API shares Flags' 600 requests a minute; /experiments/mcp allows 240 a minute. Past the limit you get 429 with Retry-After.

In an environment that requires approval, a start, ramp, stop or decide becomes a Flags approval request instead of applying, and the answer says so (flagChange.pendingApproval).

Worked example

Define a conversion metric, propose an experiment on the new-checkout flag, start it, then read its results:

BASE="$VENOM_CLOUD/experiments/api/v1"
H="Authorization: Bearer $VENOM_API_KEY"

curl -X POST "$BASE/metrics" -H "$H" -H 'Content-Type: application/json' \
  -d '{"environment": "production", "key": "purchase", "type": "conversion", "event": "checkout_completed"}'

curl -X POST "$BASE/experiments" -H "$H" -H 'Content-Type: application/json' -H 'Idempotency-Key: exp-checkout-1' \
  -d '{"flag": "new-checkout", "primary": "purchase", "hypothesis": "A one-page checkout converts better.",
       "guardrails": [{"metric": "checkout_errors", "tolerated": 0.02}]}'

curl -X POST "$BASE/experiments/<experiment-id>/start" -H "$H" -H 'Content-Type: application/json' -H 'If-Match: "1"' \
  -d '{"turnOn": true, "comment": "50/50 in production"}'

curl "$BASE/experiments/<experiment-id>/results" -H "$H"

The guardrail metric must exist too. Start, ramp, pause, resume, stop and decide answer {experiment, flagChange: {auditId, diff, pendingApproval, dryRun}, warnings}; flagChange.auditId is the Flags history entry to revert. Each answer's ETag is the experiment's version.

The same proposal over MCP, from any client connected to /workspace/mcp/exp:

{ "jsonrpc": "2.0", "id": 1, "method": "tools/call",
  "params": { "name": "exp_experiment_propose",
              "arguments": { "flag": "new-checkout", "primary": "purchase",
                             "hypothesis": "A one-page checkout converts better.",
                             "guardrails": [{ "metric": "checkout_errors", "tolerated": 0.02 }],
                             "idempotencyKey": "exp-checkout-1" } } }

Then exp_experiment_start with {"id": "<experiment-id>", "turnOn": true, "expectVersion": 1} and exp_experiment_results with {"id": "<experiment-id>"}.

Running an experiment

A typical run for an agent:

  1. exp_metric_list, then exp_metric_create for what is missing (count, sum, conversion, ratio or retention over tracked events; direction: "down" when less is better).
  2. exp_experiment_propose with the flag, the primary metric, a hypothesis and guardrails. exp_plan_get (or GET /experiments/api/v1/plan?primary=…) estimates the sample alone, saving nothing.
  3. exp_experiment_start. It is refused when the flag is off in the environment unless turnOn is true.
  4. exp_experiment_results each day. recommendation.action is ship, rollback, keep_running, inconclusive or investigate. investigate means a sample ratio mismatch: do not act on the numbers.
  5. exp_experiment_decide with decision (ship with a variation index, rollback or iterate) and a reason. It applies the Flags change, writes a decision note into Notes and sends experiments.decided.

Writes take expectVersion (or If-Match), idempotencyKey (or Idempotency-Key; a retry within 24 hours returns the first answer), comment (shown in the flag's history) and dryRun. Shares are in thousandths of a percent (50000 = 50 %), between 1000 and 99000. Ramping restarts the analysis, so ramp early and rarely. A metric an experiment uses can only be renamed.

A guarded rollout (kind: "rollout") needs at least one guardrail. Its default ladder is 1 %, 5 %, 25 %, 50 %, then 100 %, a day each. The sweep advances it, promotes it after the last step, or rolls it back and files a Tasks issue when a guardrail breaks. exp_experiment_rollout (action: "pause" or "resume") holds it; a breach still rolls a paused rollout back.

Sending events

POST /experiments/sdk/events takes {"events": [...], "exposures": [...]}: up to 1000 of each, 512 KB. An event is {event, unit: {kind, key}, value?, props?, ts?}; props is at most 2 KB of JSON. Invalid entries are dropped and counted in the 202 answer. Past the org's daily cap it answers 429 with Retry-After. See Sending events for the TypeScript SDK's track() and retention.

Events

5 org webhook events, generated from lib/webhooks/events.ts. Subscribe an endpoint to them in Settings, Webhooks (see Webhooks and events).

Event When
experiments.guardrail_breached A guardrail metric of a running experiment is worse than its tolerated regression (once per experiment and metric).
experiments.decided An experiment was decided: a variation shipped, a rollback or an iteration, by a person, an agent or the automatic guardrail rollback.
experiments.started An experiment started serving its treatments (by a person, an agent, or an approved request): the experiment, flag, environment, the treatments' share and who started it.
experiments.ramped A running experiment's treatment share changed (by hand, or a guarded rollout's next step): the share before and after, and who.
experiments.stopped A running experiment was stopped and control is served to everyone: the experiment, flag, environment and who stopped it.

The data of both events has the experiment's experiment id, name, flag, project, environment and url. experiments.decided adds decision, variation, reason, note (the decision note's path) and auditId (the flag's history entry); an automatic rollback adds automatic: true. experiments.guardrail_breached adds metric, arm, harm, tolerated, p and autoRollback, and is sent once per experiment and metric.

Experiments has no webhooks of its own. The flag changes it makes also reach Flags webhooks as flag.update.

REST routes

The Experiments API: 22 operations, generated from services/nodejs-cloud/contracts/experiments-openapi.yaml. "Key or token" is a Venom API key or an OAuth access token; SDK keys and client ids belong to an environment and are for your app's runtime.

/experiments

Method Path Summary Auth
POST /experiments/sdk/events Send events and exposures SDK key or client id
GET /experiments/api/v1/experiments List experiments key or token
POST /experiments/api/v1/experiments Propose an experiment (a draft; nothing is served yet) key or token
GET /experiments/api/v1/experiments/{id} Get an experiment key or token
PATCH /experiments/api/v1/experiments/{id} Edit a draft key or token
GET /experiments/api/v1/experiments/{id}/results Results from the daily rollups, cached until they change key or token
GET /experiments/api/v1/experiments/{id}/timeline The primary metric's cumulative lift by day, for a chart key or token
POST /experiments/api/v1/experiments/{id}/{action} Start, ramp, stop or decide an experiment (each is a Flags change as the caller) key or token

/events

Method Path Summary Auth
GET /experiments/api/v1/events The events explorer - event names with 14 days of volume, and a sample of recent events key or token

/plan

Method Path Summary Auth
GET /experiments/api/v1/plan Plan an experiment - the sample it needs and how long recent traffic takes to deliver it (nothing is saved) key or token

/query

Method Path Summary Auth
POST /experiments/api/v1/query Run a trend, funnel or retention query (read-only, time-limited; nothing is saved) key or token

/insights

Method Path Summary Auth
GET /experiments/api/v1/insights List saved insights of an environment key or token
POST /experiments/api/v1/insights Save a query as an insight key or token
GET /experiments/api/v1/insights/{id} One insight key or token
PATCH /experiments/api/v1/insights/{id} Rename an insight or change its query (If-Match with its version) key or token
DELETE /experiments/api/v1/insights/{id} Delete an insight (Notes pages that embed it show "Insight unavailable") key or token
GET /experiments/api/v1/insights/{id}/result Run a saved insight now - what a Notes page embedding it shows key or token

/metrics

Method Path Summary Auth
GET /experiments/api/v1/metrics List metrics of an environment key or token
POST /experiments/api/v1/metrics Create a metric key or token
GET /experiments/api/v1/metrics/{key} Get a metric key or token
PATCH /experiments/api/v1/metrics/{key} Update a metric (not what it measures while an experiment uses it) key or token
DELETE /experiments/api/v1/metrics/{key} Delete a metric no experiment uses key or token

MCP tools

25 tools: 12 read, 13 write. Generated from the code; "write (destructive)" is the tool's destructiveHint. A tool with a REST route is the same call over HTTP.

Tool Title Access Scope REST route
exp_event_list List events read experiments:read
exp_experiment_decide Decide an experiment write experiments:write
exp_experiment_get Get an experiment read experiments:read
exp_experiment_list List experiments read experiments:read
exp_experiment_propose Propose an experiment write experiments:write
exp_experiment_ramp Ramp an experiment write experiments:write
exp_experiment_results Experiment results read experiments:read
exp_experiment_rollout Pause or resume a rollout write experiments:write
exp_experiment_start Start an experiment write experiments:write
exp_experiment_stop Stop an experiment write experiments:write
exp_experiment_timeline Experiment lift over time read experiments:read
exp_experiment_update Change a draft experiment write experiments:write
exp_insight_delete Delete an insight write (destructive) experiments:write
exp_insight_get Read an insight read experiments:read
exp_insight_list List insights read experiments:read
exp_insight_result Run an insight read experiments:read
exp_insight_save Save an insight write experiments:write
exp_insight_update Change an insight write experiments:write
exp_metric_create Create a metric write experiments:write
exp_metric_delete Delete a metric write (destructive) experiments:write
exp_metric_get Read a metric read experiments:read
exp_metric_list List metrics read experiments:read
exp_metric_update Change a metric write experiments:write
exp_plan_get Plan a sample size read experiments:read
exp_query_run Run a trend, funnel or retention query read experiments:read

Each tool takes format: "json" for a structured answer instead of compact markdown. project defaults to the org's default Flags project and environment to production. The tool list is the same for every caller; a read-only credential is refused in words when it calls a write tool.