Alerts and notifications
Run saved log searches every 5 minutes with a signed webhook, and read the notification inbox for budget thresholds, spend forecasts, spend anomalies and key rotation.
Venom Cloud has two kinds of alert:
| Page | What fires | Who reads | Who changes |
|---|---|---|---|
Alerts (/alerts) |
a saved log query matching at least N requests in a 5-minute run | audit:read (owner, admin, developer) |
org:manage (owner, admin) |
Notifications (/notifications) |
spend thresholds, forecasts, spend jumps, key rotation and model lifecycle reminders | usage:read (every role) |
billing:manage or org:manage |
Venom Cloud sends no email. Alerts go to the console and to webhooks you run.
Alerts: /alerts
An alert is a saved search. Every 5 minutes the cron runs it over the requests logged since its last run. A run that matches at least the threshold records a hit and, if the org set a webhook, posts it there.
Create one
Either turn on a built-in suggestion with one click, or fill in the form:
| Field | Rule |
|---|---|
| Name | 1 to 100 characters |
| Query | a plain search that parses clean: no time terms (the evaluator sets the window) and no pipes. Content terms (prompt:, response:, content:) work and need content:read. Examples: has:tools tool:bash, status:5xx key:ci*, prompt:"BEGIN OPENSSH" |
| Threshold | matches in one run that make a hit; a whole number from 1 to 1,000,000, default 1 |
From Requests, Share, Create alert opens this form with the current search, its time range and pipes dropped.
Built-in suggestions
| Name | Query | Threshold |
|---|---|---|
| Secret in a prompt | has:content plus the secrets detector: a prompt holding something shaped like an API key, token or private key (sk-…, ghp_…, AKIA…, -----BEGIN … PRIVATE KEY-----) |
1 |
| Prompt blocked by a filter | filter_action:block (a prompt filter returned 403 prompt_blocked) |
1 |
| Shell tool use | tool:(bash OR shell OR *exec* OR *terminal* OR *command*) |
1 |
| Large prompt | prompt_chars:>200k |
1 |
| Error burst | status:5xx |
50 |
The page
- Alerts table: name, query (opens it in Requests), threshold, hits in the last 24 h, last run, and state: on, paused, or the last run's error. Pause, resume or delete each one.
- Hits: newest first; each count opens exactly those requests in Requests. Hits newer than your last visit are unread; the sidebar's Alerts link shows the count, and opening the page marks them seen. Hits are kept 90 days.
- Webhook: the org's URL, its signing secret's last four characters, Test, Remove.
How a run works
- Each run reads the requests logged in
(last checked, now − 60 s], at most 24 hours. The minute's lag lets sealed content land. - The watermark moves only when a run succeeds, so no request counts twice; a failed run keeps its error and retries next time. Pausing stops runs; resuming starts from now.
- Alerts that read bodies (content terms, the secrets detector) open at most 500 candidates a run, newest first; the hit says
truncatedwhen there were more. Each such run writes onecontent.alert_evaluatedaudit row, never the text. Alerts on derived fields (tool:,prompt_chars:) read no bodies. - At most 50 alerts per org. Changes are audited:
alert.created,alert.updated,alert.deleted,alert.webhook_set,alert.webhook_removed.
The webhook
The org's own http or https URL (on the hosted cloud, https to a public address only). One POST per hit, 5 s timeout, no redirects, no retries; the outcome shows on the hit. The signing secret (vwh_…) is made on the first save or on rotate, shown once, and stored sealed.
Headers:
| Header | Value |
|---|---|
venom-event |
alert.hit, or alert.test from the Test button |
venom-delivery |
the hit id |
venom-signature |
t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>">, keyed by the secret |
Body (at most 50 request ids; never a prompt, response or matched text):
{
"type": "alert.hit",
"id": "42",
"created_at": "2026-11-03T09:15:00.000Z",
"org": { "id": "…" },
"alert": { "id": "…", "name": "Shell tool use", "query": "tool:(bash OR shell)", "threshold": 1, "detector": null },
"count": 3,
"truncated": false,
"window": { "since": "…", "until": "…" },
"request_ids": ["req_…"],
"links": { "alert": "https://example.com/alerts?hit=42", "explorer": "https://example.com/requests?q=id:…", "requests": ["https://example.com/requests/req_…"] }
}Check the signature like a Stripe webhook: recompute, compare in constant time, refuse a t older than 5 minutes.
import { createHmac, timingSafeEqual } from "node:crypto";
export function verify(body: string, header: string, secret: string): boolean {
const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
const t = Number(parts.t);
if (!t || Math.abs(Date.now() / 1000 - t) > 300) return false;
const want = createHmac("sha256", secret).update(`${t}.${body}`).digest("hex");
const got = String(parts.v1 ?? "");
return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want));
}Links use the deployment's VENOM_CLOUD_ORIGIN, else the cron's own origin.
Notifications: /notifications
The inbox behind the top bar's bell, newest first. Items newer than your last visit are unread; opening the page marks them seen.
| Kind | When | Its button |
|---|---|---|
| Threshold | spend passed a threshold of a budget or of the monthly spend cap; once per threshold and window | Open budgets, or Open usage for the cap |
| Reached | a threshold of 100 % or more | the same |
| Forecast | at the current rate, a budget or the cap runs out before its day or month ends; once a window | the same |
| Spend anomaly | a key, member, model or gear spent far more in an hour than its usual for that hour of day | opens its evidence page |
| Key rotation | a provider key is due for rotation, or was retired | Rotate the key, or Open the provider |
| Lifecycle | a model deprecation or an SLA event | Open model lifecycle, or Open SLAs |
Alert settings
Thresholds and forecasts apply to every budget and to the monthly spend cap; a changed limit or a new day or month starts over. Spend jumps need no budget. Checked every 5 minutes. Changing them needs billing:manage or org:manage.
| Setting | Default | Rule |
|---|---|---|
| Spend alerts | on | off writes nothing to the inbox or to webhooks for spend |
| Thresholds (% of the limit) | 50, 80, 100 | comma separated, 1 to 200, at most 6 |
| Alert when the forecast crosses a limit | on | |
| Alert on sudden spend jumps | on | |
| Ignore jumps under (USD an hour) | 10 | above the usual spend; keeps small keys quiet |
| Sensitivity | Medium: 6 spreads and 3× the usual | Low: 8 spreads and 5×; High: 4 spreads and 2× |
Where they go
Notifications always land in this inbox. To get them elsewhere, add an org webhook under Settings, Webhooks (/settings/webhooks) that takes spend.threshold, spend.forecast or spend.anomaly: signed with HMAC-SHA256, retried, with a delivery log and a test button. For mail or chat, point a webhook at a relay you run (Slack, Teams or your mail gateway).
A spend anomaly: /notifications/anomalies/{id}
The evidence for one spend jump and what to do about it:
- Figures: spent in the hour, the usual for that hour, the threshold, how many times the usual.
- Last 48 hours: the spend against the usual hour-of-day curve.
- Who spent it: the top keys, each with Pause or Resume. Pausing needs
keys:writeorbilling:manage. - Top requests: each opens its trace (needs
logs:read). - Also jumped: other keys, members, models or gears that jumped in the same hour.
- How this was detected: the method and the sensitivity used.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
400 invalid_query with alert_window or alert_pipeline |
Remove time terms and pipes from an alert's query. |
409 too_many_alerts |
The org has 50 alerts; delete one. |
| An alert shows an error badge | Its last run failed; hover for the reason. It retries on the next run. |
| A hit shows a webhook error | The POST failed or timed out after 5 s. Hits are not retried; use Test after fixing the endpoint. |
| No spend notifications | Spend alerts are off, no budget or cap is set (thresholds and forecasts need one), or the jump was under the USD floor. |
Related
- Log query language
- Requests and request traces
- Audit log: the
alert.*andcontent.alert_evaluatedrows.