Alert thresholds and cooldowns

shield-alerting holds the rules, evaluates them against a rolling hour of your events, dispatches notifications, and keeps a history you can acknowledge.

Who can use it. A tenant admin with a dashboard session, for every action. No plan gate. The function is console-only: it is absent from the worker allowlist, so there is no public URL, and the dashboard calls it through supabase.functions.invoke. pg_cron is the other caller, using a cron secret for the evaluate action only.

Actions: evaluate, list_rules, create_rule, update_rule, delete_rule, test_channel, list_history, acknowledge_alert. Every call needs tenant_id in the body; without it the answer is 400 Missing tenant_id.

Evaluation runs on its own. The pg_cron job shield-alerting-evaluate-5m, added by migration 20260903211000, fires every five minutes and sweeps every tenant with at least one active rule. Calling evaluate yourself is an immediate extra check, never a requirement.

Evaluation computes five metrics for the last hour with a single in-database aggregate (shield_alerting_hour_metrics). No row is read into the function and counted there, so a busy hour is counted exactly.

Two things to know before you set a threshold. response_ms is the full request including awaited outbound actions, so p95_latency_ms moves when a third party you dispatch to is slow. And unlike the Observability tiles, this aggregate does not exclude test-mode signals, so sandbox traffic and dashboard test signals count toward events_per_hour and can fire a live rule.

There is no pre-aggregation layer: every evaluate recomputes the window.

const { data, error } = await supabase.functions.invoke("shield-alerting", {
  body: {
    action: "create_rule",
    tenant_id: "<uuid>",
    name: "High risk spike",
    metric: "high_risk_count",
    operator: ">",
    threshold: 10,
    severity: "critical",
    channel_type: "pagerduty",
    channel_config: { routing_key: "R0123456789ABCDEF" },
    cooldown_minutes: 30,
  },
});

The response is {"status":"created","rule":{"id":"...","name":"High risk spike"}} and an alerting.rule_created audit entry is written.

Confirm it worked. Dashboard > Observability > Alert rules lists the new rule with its channel badge, severity, condition and cooldown.

One evaluate call, or one cron tick, does this for the tenant:

1. Read active rules from shield_alert_rules, paged explicitly in blocks of 1,000 up to 5,000 per tick, least recently evaluated first (last_evaluated_at, stamped on every rule a tick evaluates). Hitting that ceiling sets truncated: true on the response rather than silently dropping rules, and the next tick reads the rules this one did not reach.
2. Compute the five rolling-hour metrics.
3. For each rule, apply the operator to the metric value against the threshold. An operator the function does not recognise evaluates to no breach.
4. On a breach, claim the cooldown window: a conditional update that sets last_fired_at and increments fire_count only if last_fired_at still holds the value this run read. A rule inside its cooldown is skipped; a concurrent tick that loses the claim skips the firing entirely, so one breach never notifies twice.
5. Insert the shield_alert_history row. If that insert fails, the claim is released so the next tick can retry rather than swallowing the alert.
6. Dispatch to the channel, recording dispatch_ok and, on failure, dispatch_reason.
7. Write one alerting.rules_evaluated audit entry if at least one alert fired.

A dispatch failure does not undo the firing: the history row stands and the cooldown has started, so an unreachable channel loses notifications rather than replaying them later.

{
  "status": "ok",
  "tenant_id": "<uuid>",
  "rules_evaluated": 5,
  "alerts_fired": 1,
  "fired_alerts": [
    {
      "rule_id": "<uuid>",
      "rule_name": "High risk spike",
      "metric": "high_risk_count",
      "operator": ">",
      "threshold": 10,
      "current_value": 14,
      "tenant_id": "<uuid>",
      "severity": "critical",
      "fired_at": "2026-09-16T14:22:00.000Z",
      "dispatch_ok": false,
      "dispatch_reason": "http_404"
    }
  ],
  "metrics": {
    "events_per_hour": 342,
    "p95_latency_ms": 87,
    "high_risk_count": 14,
    "error_rate": 0.012,
    "mean_risk_score": 0.45
  },
  "truncated": false
}

Update a rule. Only these fields can be changed: name, metric, operator, threshold, severity, channel_type, channel_config, cooldown_minutes, is_active. Anything else in the body is ignored, so fire_count, created_by and last_fired_at cannot be written from outside.

{
  "action": "update_rule",
  "tenant_id": "<uuid>",
  "rule_id": "<uuid>",
  "threshold": 20,
  "cooldown_minutes": 60,
  "is_active": false
}

The reply is {"status":"updated","rule_id":"..."}. A body with no recognised field answers 400 No valid fields to update, and an invalid value answers with the same message as rule creation.

Delete a rule. {"action":"delete_rule","tenant_id":"...","rule_id":"..."} answers {"status":"deleted","rule_id":"..."}. The rule's history rows stay.

List rules. {"action":"list_rules","tenant_id":"..."} returns {"rules":[...]}, active and inactive, newest first. Since 2026-09-18 a rule's secrets (webhook_url, routing_key, api_key) are stored encrypted in the platform vault and never returned: channel_config in this answer carries a hint per secret (a URL's origin, a key's last four characters) and stored, which is vault once the row has been written that way. To test a stored rule, call test_channel with its rule_id.

Rule updates and deletions are not audit-logged today; creation and acknowledgement are.

List history.

{ "action": "list_history", "tenant_id": "<uuid>", "limit": 50 }

limit defaults to 50 and is capped at 200. The reply is {"history":[...]}, ordered by fired_at descending. Each entry carries id, rule_id, metric_value, severity, channel_type, payload, acknowledged_at, acknowledged_by and fired_at. The payload holds the rule name, metric, operator, threshold and current value as they were at firing time, so a later edit to the rule does not rewrite history.

Acknowledge.

{ "action": "acknowledge_alert", "tenant_id": "<uuid>", "alert_id": "<uuid>" }

Sets acknowledged_at and acknowledged_by to the calling user and answers {"status":"acknowledged","alert_id":"..."}. An unknown id answers 404 Alert not found; an entry already acknowledged answers 409 Already acknowledged. The action writes an alerting.alert_acknowledged audit entry.

Confirm it worked. Dashboard > Observability > Alert history, with the Pending filter: the entry moves out of that list and shows an Acknowledged badge.

Start wide and tighten. A rule that never fires teaches you nothing. Set a first threshold you expect to trip occasionally, watch fire_count in the rules list for a week, then raise it.

Match the cooldown to the response. The cooldown is the minimum gap between two firings of the same rule, and the default is 15 minutes. A page-someone rule is usually 5 to 10 minutes; an informational Slack rule is better at 30 to 60, since evaluation happens every five minutes and a sustained breach would otherwise notify twelve times an hour.

Layer two rules on one metric. One at a lower threshold with severity: "warning" to Slack, one at a higher threshold with severity: "critical" to PagerDuty. Both fire independently, each with its own cooldown.

Test the channel before you need it. Dispatch failures are reported in fired_alerts[].dispatch_reason, but only after a real breach. test_channel tells you now, and reports the real outcome rather than a hopeful 200.

Watch for test traffic. Because the evaluation aggregate counts test-mode signals, a burst from the dashboard test button or a sandbox key can trip events_per_hour and error_rate. If that matters, favour high_risk_count and mean_risk_score, which test signals rarely move.