Skip to content
Docs menu

The loop

An error spike opens an incident, Venom ranks what changed, rolls back the flag that did it through Venom Flags (if you allow it), and files a Tasks issue for an agent with the stack and the suspects.

View .md

A bad flag change goes out. Errors climb. Within a minute or two you have an incident, the flag that caused it is off again, and an agent has a Tasks issue with the stack and the commits to look at. This page is how that works and how to turn the risky part on.

What happens, in order

  1. An alert rule with the action Open incident fires on an error issue (a spike or a threshold). The incident is declared at the rule's severity and linked to the issue.
  2. Venom ranks what changed in the two hours before the spike: flag changes in the same project and environment (read from the Flags audit log) and releases. The timeline gets a Likely cause entry with the top three.
  3. If the environment allows it and the top candidate is a flag with enough confidence, the change is reverted.
  4. A Tasks issue is filed in the team you mapped, with the error issue, a compact stack, the suspect, the rollback result and the pull requests and commits Tasks has linked around the release. It is delegated to the triage agent you chose.
  5. When the incident resolves, a postmortem draft is written to Notes.

Every step is a timeline entry, with its actor (incidents:auto for the automatic ones) and the channel auto. Running the rule again does nothing twice: one incident per issue while it is open, one rollback, one Tasks issue.

How the ranking works

A flag change scores on two things.

  • Timing. It landed shortly before the spike. The score falls from 1 at the spike to 0 two hours earlier.
  • Concentration. Among the issue's sampled events that carry the flag, how many were served one value, compared with that value's share of all the flag's evaluations. If every error saw true and true is served to 5% of evaluations, the flag is almost certainly it. If the errors split the way the traffic does, it is not.

The confidence is 30% timing and 70% concentration. A change with no sampled error carrying its flag is capped at 35%: worth a look, never enough to act on. Releases score on timing, on the share of sampled errors on that release, and on whether the issue first appeared after it was deployed. A release is shown but never rolled back; there is nothing in Venom Cloud to revert a deploy.

You can read the ranking at any time: the Likely cause panel on the incident page, GET /incidents/api/v1/incidents/{ref}/implicate, or the incidents_incident_implicate MCP tool.

Automatic rollback

It is off by default. In Incidents, Settings, Close the loop (an owner or admin), switch Roll back by itself on for an environment and pick the confidence it needs (50% to 90%, 70% by default). With it off, the timeline says so and the incident page shows a Roll back button on the flag candidate, and agents can call incidents_incident_rollback.

A rollback is a normal Venom Flags change:

  • It is FlagsService.revert of the change that was implicated, so it is audited in Flags under the actor incidents:auto (or the person or agent who pressed the button), and the original entry is marked reverted.
  • It is itself revertible from the Flags audit log.
  • If the flag moved on since (so the revert is refused) and it is a permanent flag, Venom turns it off in that environment instead. A temporary flag is left alone and the timeline says it could not be rolled back.
  • Approvals are respected. In an environment that requires approvals, Venom files a request to turn the flag off and stops. A reviewer approves it in Venom Flags.

The Tasks issue

In the same settings page, give the project a Tasks team key and a triage agent (mcp:…, key:…, daemon:… or session:…, the ids Tasks delegates to). Empty means the first team and no delegate. The issue is created by the system actor, and if you run an intake runner for that team, the usual intake path can take it to a pull request.

The body has what an engineer needs to start: counts and users, the stack with in-app frames first, the flags served, the ranked suspect with its reasons, what the rollback did, suspect commits, and a reproduction hint.

For agents

Tool Does
incidents_incident_implicate The ranking, with evidence
incidents_incident_rollback The rollback, with ref for an incident's candidate or with project, flag, auditId and env for any Flags change (a short postmortem is written when no incident is behind it)

See Build on Venom Incidents. Over REST the same actions are GET /incidents/api/v1/incidents/{ref}/implicate, POST …/rollback, POST /incidents/api/v1/rollback and GET/PUT /incidents/api/v1/settings.