Skip to content
Docs menu

Error alerts

Rules that tell you about a new issue, a regression, a spike, a threshold or a burning error budget: the bell, org webhooks and Slack, a dry run over the last 24 hours, and how a hit is decided.

View .md

Alert rules are in Incidents → Alerts. They run in the sweep cron every minute over the hourly counts, never on the path errors take in, so they cost the ingest nothing. Seeing and testing rules is for every member; creating, changing and deleting them needs an owner or admin (incidents:admin).

Rules

A rule belongs to a project and, optionally, one environment. It has a kind, an action, a cooldown and an optional Slack channel.

Kind Fires when
New issue An issue with a cause not seen before appears. Once per issue.
Regression A resolved issue happens again where it should not. Once per regression.
Spike An issue's events this hour are factor times (5 by default) its usual rate for this hour of the day, and at least minEvents (10).
Threshold An issue has at least N events in the last 1, 6 or 24 hours, or at least N users affected (and an event in the last hour).
Error budget burning The project's errors spend the budget of a service faster than its target allows.

Spike. The baseline is the spend anomaly model's, reused as it is: the median and spread of the same hour over the last 14 days, or of the last 48 hours while the issue is young, and a jump past that. An issue with no history is not a spike (it is a new issue); it counts only past minEvents x factor events in the hour. Sensitivity is low, medium (default) or high.

Error budget. Give the requests an hour the service sees and the share that must succeed (99.9 % is 0.999). The budget is the errors an hour that target allows, requestsPerHour x (1 - target). The burn rate is the errors an hour over that budget, as in the daemon SLOs: a rule fires when the last hour burns at 6x or more and the last six hours burn at 3x or more, so one noisy hour does not alert and a slow leak does not hide.

Cooldown. The same issue does not alert again for the same rule inside its cooldown (60 minutes by default). A project-wide rule (the error budget) has one cooldown for the project.

What a hit does

  1. It is recorded (rule, issue, title, the numbers behind it) and shown under Recent hits and in the bell (app Incidents, unread until you open Incidents).
  2. An errors.alert.fired event goes to the org's webhooks: rule, kind, project, issue id, title, numbers and the action. Never an event payload, a stack or a user.
  3. If the rule has a Slack channel and the org's Slack app is connected, the app posts the title, the numbers and a link. The bot must be in the channel. A failed post is logged; the hit still stands.

Separately from rules, every new and every regressed issue produces errors.issue.created and errors.issue.regressed for the org's webhooks (id, project, title, count, release), within a few minutes.

Open an incident declares an incident at the severity you choose (SEV3 by default), linked to the issue, once while one is open for it; a project-wide rule (error budget) opens one per rule. Page the on-call (and an opened incident) pages the org's default escalation policy: the same alert again is the same page.

Test a rule

Test rule in the dialog runs the rule over the last 24 hours, hour by hour, and lists what it would have fired. Nothing is recorded or sent. Over MCP, incidents_alert_set with test: true does the same.

REST

GET, POST /incidents/api/v1/alerts, PUT, DELETE /incidents/api/v1/alerts/{id} (If-Match: "<version>" guards a replace), POST /incidents/api/v1/alerts/test, GET /incidents/api/v1/alerts/hits and POST /incidents/api/v1/alerts/seen. A rule body is { "project", "env", "kind", "params", "action", "cooldownMin", "slackChannel", "name", "enabled" }.

{ "project": "web", "env": "production", "kind": "spike", "params": { "factor": 5, "minEvents": 20 }, "action": "notify", "slackChannel": "C0123ABCD" }