Venom Experiments
A/B tests on Venom Flags flags with sound statistics (CUPED, always-valid sequential results, sample ratio checks, guardrails), run by people or agents, with a decision note left in Notes.
Venom Experiments runs controlled experiments on your own app. An experiment sits on a Venom Flags flag: the flag's rollout decides who sees what, your app sends the events that say what happened (sending events), and Experiments does the statistics. An agent can propose an experiment, start it, read the results each day and ship the winner; everything it does to the flag is in the flag's history and can be reverted there.
The idea in one paragraph
An experiment never assigns anyone itself. Starting it puts a percentage rollout, marked with the experiment's id, on a
flag's fallthrough (or on one rule). The Flags evaluator then answers with a reason that says inExperiment, the SDK
posts an exposure for each unit the first time it sees it, and from then on the unit counts in the arm it was
served. Because the SDK, the server and the analysis share one evaluator and one bucketing, they agree on who saw
what.
Metrics
A metric belongs to a Flags environment and reads the events your app tracks:
| Type | What each unit contributes |
|---|---|
count |
the number of its events after exposure |
sum |
the value of its events, summed |
conversion |
1 if it had at least one event, else 0 |
retention |
1 if it had an event exactly retentionDays after exposure (only units old enough are counted) |
ratio |
the ratio of two other metrics (revenue per order), with the delta method for the variance |
Each metric has a direction (which way is good) and, for count and sum, an optional winsorize cap so a few
extreme units do not decide the result. A unit with no events counts as zero. Events from the day of the exposure on
count; earlier days are the CUPED covariate.
An experiment
An experiment names its flag, its control and treatment variations, the share of units in the treatments (50 % by default), a primary metric, optional secondary metrics and guardrails, and how to analyse:
- Sequential (the default). Results use the mixture sequential probability ratio test: the p-value and the interval are valid however often you look and whenever you stop, so you can read them every day.
- Fixed horizon. Results wait for the planned sample per arm; before that they show estimates and no p-value.
The plan comes from the last 14 days of your traffic: the sample per arm that detects a relative change (mde, 5 % by
default) at the chosen power, and the days it takes. For a sound baseline, track an event for every visit or session:
the busiest event of the environment is taken as the population. Without history the plan is a stated default of
5,000 units per arm.
CUPED (on by default) adjusts each unit's outcome by the same metric in the 14 days before exposure. It does not change the estimate and narrows the interval when past behaviour predicts the outcome.
Secondary metrics are corrected together with Holm's method. A guardrail is a metric that must not get worse by
more than a tolerated share of control's mean (2 % by default), tested one-sided and sequentially. A breach emits the
experiments.guardrail_breached org webhook event once. With autoRollback (off by default) it also serves control
through a Flags change made by the actor experiments:guardrail, which is audited and revertible.
Sample ratio mismatch
Every analysis compares the units in each arm with the planned split by a chi-square test. When p is below 0.001 something in assignment or logging is broken (an SDK that exposes only in one arm, a redirect that drops users). The numbers are then hidden behind a warning and the recommendation is "investigate".
Changing the split
ramp changes the treatments' share through Flags. A new split means units can move arms, so the analysis starts over:
results count only the units exposed after the change. Ramp early and rarely.
The life of an experiment
draft → running → stopped or decided.
- Propose creates the draft and plans the sample. Nothing is served.
- Start makes the Flags change. The flag must be on in the environment (or pass
turnOn). In an environment that requires approvals the change becomes a request; the experiment starts when it is applied. - Stop serves control to everyone. A stopped experiment can still be decided.
- Decide is
ship(serve one variation to everyone),rollback(serve control) oriterate(serve control and note that another round follows). It applies the Flags change, writes a decision note into the organisation's Notes vault underExperiments/(hypothesis, setup, results table, the recommendation at the time, links to the flag's audit entry) and emitsexperiments.decided.
Only one experiment runs on a flag, in an environment and on a rule, at a time.
Starting, ramping and stopping are org webhook events too: experiments.started (also when an approved request starts
it), experiments.ramped (by hand, or a guarded rollout's next step, with the share before and after) and
experiments.stopped. Each carries the experiment, flag, environment, the treatments' share and who did it. Subscribe
in Settings → Webhooks (Webhooks and events).
Insights
Trends, funnels and retention over the same events, saved insights, and live chart blocks for Notes pages are on the Insights page; see Insights and dashboards.
Guarded rollouts
A guarded rollout is an experiment without a hypothesis: a flag change that ramps through steps while guardrail metrics
hold, and rolls back by itself when one breaks. Tick Guarded rollout in the new-experiment flow, or propose with
kind: "rollout" over REST or MCP.
- Steps are a share of units and the least time to stay there. The default is 1% → 5% → 25% → 50% (a day each); 100%
follows the last step. Pass your own
steps: [{ share, minHours }](shares in thousandths of a percent, increasing). - At least one guardrail is required (a metric and how much worse than control it may get). The analysis is sequential, so the guardrails can be read at any time without inflating false alarms.
- The sweep (every few minutes) moves each running rollout. When the step's minimum time has passed and the guardrail
tests have enough units and show no breach, it ramps to the next share. After the last step it ships the treatment to
everyone. Each move is a Flags change by the system actor
experiments:rollout: audited, revertible, and a request in an environment that requires approval (the rollout waits for it; a turned-down request pauses the rollout). - A breach serves control at once (even from a pause), marks the rollout
Rolled back, raisesexperiments.guardrail_breachedandexperiments.decided, writes the Notes decision note and files a Venom Tasks issue with the evidence when the organisation has Tasks. - Pause and resume hold the ramp (a button on the experiment page,
POST …/pause,…/resume, or theexp_experiment_rollouttool). Resuming restarts the step's minimum time. A rollout moves by its steps, so a manual ramp is refused.
The experiment page shows the step ladder, where it is, what it is waiting for and each guardrail's state. Rollouts are off until you start one; nothing about existing experiments changes. Canaries of deploys that are not behind a flag are a separate matter.
The app
Experiments is an app in the workspace rail (press G then X). It lives at /exp (the console's /experiments page is
the separate gear A/B test page) and needs the Experiments add-on, which includes Venom Flags. While billing is off, or
the add-on has no price, every organisation has it.
- Experiments lists the project's experiments in the environment in view: status, the primary metric's lift with
its interval drawn against zero, days running, and a badge when the sample ratio check failed. Press
/to filter,jandkto move, Enter to open andcfor a new one. The status tabs count each state. - New experiment (
c, or ⌘K, "New experiment") picks a flag and which variations are the arms, takes a name and a hypothesis, a primary metric, secondary metrics and guardrails (with how much regression you tolerate), the analysis (sequential or fixed horizon), the treatments' share, the smallest change worth detecting and the power. The plan at the bottom updates as you change them: the units each arm needs and how many days recent traffic takes to deliver them. Saving makes a draft; nothing is served until you start it. - An experiment's page (
/exp/{id}) starts with the verdict: the recommendation (ship, roll back, keep running, inconclusive, investigate) in words, readiness and, for a fixed-horizon test, how far it is toward its planned sample. A sample ratio mismatch shows as a red banner and hides the numbers. Below: the arms with their planned and observed traffic; the results table with each metric's lift, interval bar, evidence and verdict, and a CUPED switch that recomputes the numbers; the cumulative lift chart; guardrails (holding or breached); the flag's recent history from Venom Flags' audit log; and the experiment's own log. - Start, Ramp, Stop and Decide are buttons on the page. Each is a Venom Flags change made as you, so it is in the
flag's history and can be reverted there; in an environment that requires approvals it becomes a request and the page
says so. A ramp warns that the analysis starts over. Decide offers each treatment to ship, roll back and iterate,
with the recommended one first, takes a reason, and writes the decision note into Notes. A decided experiment links to
that note and to the flag's history. Press
rto recompute and Esc to go back to the list. - Metrics (
/exp/metrics,⇧M) is the catalog of the environment: create and edit count, sum, conversion, ratio and retention metrics, with the direction and an optional cap per user. What a metric measures can't change while a running or stopped experiment uses it, and a metric an experiment uses can't be deleted. - Events (
/exp/events,⇧E) lists the event names the environment received in the last 14 days with a volume bar for each and when it was last seen, and the newest raw rows of all events or of one name, so you can check thattrack()arrives before you build a metric on it. - Connect an agent (
/exp/agents) links to the Connect page (/connect), and to the Experiments-only endpoint under its Advanced section.
Running experiments also appear on Home with their lift, and a guardrail breach (or a decision in the last three days) shows in the bell.
Where to go next
- Sending events and exposures from your app.
- The MCP server for agents.
- The REST API.