Observability: metrics, alerting and the audit log

PasskeyBridge writes a row to shield_events for every signal it handles and a row to shield_audit_log for every administrative action. Four surfaces read those tables, all of them scoped to one tenant.

Who can use it. No plan gate: the code gates metrics and alerting on the tenant admin role rather than on a plan. The dashboard reaches both functions with supabase.functions.invoke and your signed-in session. shield-alerting is console-only; its name is absent from the worker allowlist, so https://api.passkeybridge.io/v1/shield-alerting answers 404 with x-pb-reason: unknown-function. shield-metrics has one lane an API key can reach, the Prometheus exposition, described under Prometheus export.

Where the numbers come from. Every count, percentile and breakdown is computed inside the database by service-role RPCs (shield_metrics_summary, shield_metrics_anomaly_stats, shield_metrics_timeseries, shield_metrics_latency, shield_alerting_hour_metrics). No handler loads event rows and counts them in JavaScript, so a busy window is never silently capped near a thousand rows.

Test signals. The metrics RPCs exclude rows with test_mode = true, so sandbox and test-mode traffic stays out of the tiles, the anomaly feed and the latency panel. The alert evaluator's RPC does not exclude them: a burst of test signals can fire a live alert rule.

shield-metrics computes tenant metrics on demand. Five actions: summary, timeseries, latency, anomalies and prometheus.

Who can call it. Four of the five actions take a signed-in tenant admin: the function checks the JWT and then is_tenant_admin for the tenant_id you name. The fifth, prometheus, also accepts an API key carrying metrics or events, arriving through api.passkeybridge.io. A key presented with any other action is refused with 403.

Where the parameters go. action, tenant_id and hours are read from a JSON body or from the query string, and the body wins where it carries the key. The dashboard posts a body; a Prometheus scrape is a GET with no body, so it uses the query string.

const { data, error } = await supabase.functions.invoke("shield-metrics", {
  body: { action: "summary", tenant_id: "<uuid>", hours: 24 },
});

The summary response. latency and risk are null when no row in the window carries a value for them; sample_info is kept for shape compatibility and now always reports the exact total with *_truncated: false.

{
  "window_hours": 24,
  "tenant_id": "<uuid>",
  "sample_info": {
    "events_sampled": 1482,
    "events_total": 1482,
    "events_truncated": false,
    "intelligence_sampled": 12,
    "intelligence_total": 12,
    "intelligence_truncated": false,
    "delegates_sampled": 4,
    "delegates_total": 4,
    "delegates_truncated": false,
    "negotiations_sampled": 0,
    "negotiations_total": 0,
    "negotiations_truncated": false
  },
  "events": {
    "total": 1482,
    "by_type": { "sim_swap": 47, "port_out": 12 },
    "by_result": { "success": 1475, "failure": 7 }
  },
  "latency": { "p50": 118, "p95": 402, "p99": 913, "avg": 164, "min": 22, "max": 2140 },
  "risk": { "mean": 0.21, "p50": 0.1, "p95": 0.82, "high_risk_count": 63, "critical_count": 4 },
  "intelligence": { "total": 12, "by_type": { "behavioral": 12 }, "hallucination_flags": 0, "human_reviewed": 3 },
  "agents": { "active_delegates": 4, "avg_trust_score": 0.86, "low_trust_count": 0 },
  "negotiations": { "total": 0, "by_status": {} },
  "generated_at": "2026-09-16T12:00:00.000Z"
}

high_risk_count counts events with risk_score above 0.8 and critical_count those above 0.95. low_trust_count counts active delegates with a trust score below 0.4. Latency percentiles come from shield_events.response_ms, the whole request including any outbound playbook action.

Confirm it worked. Open Dashboard > Observability. The Events tile shows the same events.total for the last 24 hours.

latency answers the Decision latency panel in one round trip. It separates two measures that are easy to confuse.

• decision_ms is PasskeyBridge's own path: authenticate, resolve the tenant, hash the identifier, match a playbook, decide. Rows written before the column existed carry null and are counted in decision.unmeasured.
• response_ms is the full request, including time awaited on Slack, your callback endpoint and email delivery.

const { data } = await supabase.functions.invoke("shield-metrics", {
  body: { action: "latency", tenant_id: "<uuid>", hours: 24 },
});

{
  "events": 1482,
  "response": { "samples": 1482, "p50": 118, "p95": 402, "p99": 913, "avg": 164, "min": 22, "max": 2140 },
  "decision": { "samples": 1204, "p50": 21, "p95": 44, "p99": 61, "avg": 25, "min": 7, "max": 320, "unmeasured": 278 },
  "histogram": { "response": [12, 88, 240, 501, 410, 180, 51], "decision": [402, 511, 230, 48, 12, 1, 0] },
  "window_hours": 24,
  "since": "2026-09-15T12:00:00.000Z"
}

Both histograms have seven fixed buckets with the same edges, counted over events rather than over time buckets. Rows with test_mode = true are excluded from every figure here.

timeseries returns event counts in fixed-width buckets for charting.

{
  "series": [
    { "timestamp": "2026-09-16T11:00:00.000Z", "event_count": 62, "avg_latency_ms": 141, "max_risk_score": 0.87 }
  ],
  "bucket_minutes": 60,
  "bucket_minutes_requested": 60,
  "total_events": 1482,
  "generated_at": "2026-09-16T12:00:00.000Z"
}

A response never carries more than 3,000 buckets. When the requested width would produce more, the function widens the bucket and reports both bucket_minutes (what you got) and bucket_minutes_requested (what you asked for). The database still aggregates every row in the window, so nothing is dropped by the widening.

The prometheus action renders the event counter and the latency summary in Prometheus text exposition format, served as Content-Type: text/plain; version=0.0.4; charset=utf-8.

# HELP passkeybridge_events_total Total identity threat events
# TYPE passkeybridge_events_total counter
passkeybridge_events_total{type="sim_swap",tenant="<uuid>"} 47
passkeybridge_events_total{type="port_out",tenant="<uuid>"} 12
# HELP passkeybridge_latency_ms Response latency in milliseconds
# TYPE passkeybridge_latency_ms summary
passkeybridge_latency_ms{quantile="0.5",tenant="<uuid>"} 118
passkeybridge_latency_ms{quantile="0.95",tenant="<uuid>"} 402
passkeybridge_latency_ms{quantile="0.99",tenant="<uuid>"} 913
passkeybridge_latency_ms_sum{tenant="<uuid>"} 243048
passkeybridge_latency_ms_count{tenant="<uuid>"} 1482

The latency block is emitted only when at least one event in the window carries response_ms. Label values are escaped for backslashes, quotes and newlines. The figures are the same aggregates the summary action returns.

How to scrape it. Present an API key carrying metrics or events in x-pb-api-key and name the tenant in the query string. The request has to arrive through api.passkeybridge.io; a direct call to the functions host answers 403 Direct access denied.

curl -s -H 'x-pb-api-key: pb_live_your_key' 'https://api.passkeybridge.io/v1/shield-metrics?action=prometheus&tenant_id=<uuid>&hours=24'

A Prometheus scrape config for the same thing:

scrape_configs:
  - job_name: passkeybridge
    scheme: https
    metrics_path: /v1/shield-metrics
    params:
      action: [prometheus]
      tenant_id: ["<uuid>"]
      hours: ["24"]
    http_headers:
      x-pb-api-key:
        values: ["pb_live_your_key"]
    static_configs:
      - targets: ["api.passkeybridge.io"]

http_headers is part of the shared HTTP client configuration and is not in every Prometheus release; check your own version's configuration reference for it. On a release that lacks it, or on a collector that cannot set a custom request header at all, put something you run in front that adds it.

Read the counter as a window, not a total. passkeybridge_events_total is declared a counter, and every scrape recomputes it over the last hours hours, so it falls as events age out of the window. That breaks the monotonicity rate() and increase() assume. Pick a window comfortably longer than the range you query and read the series as a level, or derive rates from passkeybridge_latency_ms_count over the same caveat. hours defaults to 24 and is capped at 720.

What the key does not open. Only prometheus. A key sent with summary, timeseries, latency or anomalies gets 403, because those actions return the JSON the Observability tab renders and stay on the dashboard session lane. The key must belong to the tenant in tenant_id, and the call counts against that tenant's quota like any other key call.

Until 2026-09-17 no scrape was possible: shield-metrics was absent from the worker allowlist, so the public host answered 404, and the function read its parameters from a JSON body that no scraper sends.

The anomalies action runs four detectors over the window and returns what tripped. Each detector has a minimum sample count below which it stays silent, so a quiet tenant does not generate noise.

Fewer than five events in the window short-circuits everything: the response carries an empty anomalies array, insufficient_data: true, and the event count in events_analyzed.

{
  "anomalies": [
    {
      "type": "velocity_spike",
      "severity": "critical",
      "metric": "events_per_hour",
      "description": "412 events in hour 2026-09-16T09 (mean: 58.0, +4.7σ)",
      "detected_at": "2026-09-16T09:00:00Z",
      "value": 412,
      "threshold": 180
    }
  ],
  "events_analyzed": 1482,
  "window_since": "2026-09-15T12:00:00.000Z",
  "generated_at": "2026-09-16T12:00:00.000Z"
}

Results are sorted with critical entries first. These thresholds are fixed in the function; they are not configurable per tenant. For thresholds you choose yourself, use an alert rule.

Confirm it worked. Dashboard > Observability, the Anomalies card, with the same window selected.

shield-alerting evaluates threshold rules against a rolling hour of your events and dispatches a notification when one breaks. Full detail is in Alert thresholds and cooldowns.

Who can call it. A signed-in tenant admin, through supabase.functions.invoke("shield-alerting"), for every action. pg_cron also calls the evaluate action with a cron secret and no JWT. The function is not on the worker allowlist, so there is no public URL for it.

Evaluation is already scheduled. The pg_cron job shield-alerting-evaluate-5m (migration 20260903211000) posts {"action":"evaluate"} every five minutes and sweeps every tenant that has at least one active rule. You do not have to call evaluate yourself; the dashboard button exists for an immediate check.

Actions: evaluate, list_rules, create_rule, update_rule, delete_rule, test_channel, list_history, acknowledge_alert.

What one tick does, per tenant:

1. Read active rules, paged explicitly, up to 5,000 per tick, least recently evaluated first; a tick that hit the cap reports truncated: true rather than implying it saw everything, and stamps what it read so the next tick starts with the rest.
2. Skip quarantined tenants, so a cut-off tenant's receivers are not notified on its behalf. Billing status is deliberately not a filter.
3. Fetch the rolling-hour metrics with one in-database aggregate.
4. Compare each rule's metric against its threshold.
5. Claim the cooldown window with a conditional update on last_fired_at, so two overlapping ticks cannot both notify for one breach. The loser of the claim skips the firing entirely.
6. Insert the shield_alert_history row. If that insert fails, the claim is released so the next tick retries.
7. Dispatch to the channel and record dispatch_ok, with dispatch_reason when it failed. A failed dispatch still counts as fired, because the history row exists and the cooldown has started.
8. Write one alerting.rules_evaluated audit entry if anything fired.

Confirm it worked. Dashboard > Observability > Alert history shows the firing with its severity, channel and metric value, and an Acknowledge button.

Four channel types are accepted. Any other value is rejected at rule creation with 400 Invalid channel_type. Valid: slack, pagerduty, opsgenie, webhook.

Slack posts one section block and one context block, with a bracketed severity label ([CRITICAL], [WARNING], [INFO]), the rule name, the metric, the current value and the threshold. No emoji.

PagerDuty sends an Events API v2 trigger with summary of the form [PasskeyBridge] {rule_name}: {metric} = {value}, source: "passkeybridge", component set to the metric name, and the whole alert payload in custom_details. Severity maps critical to critical and everything else to warning. A webhook_url in the config is ignored; a missing routing_key fails the dispatch.

OpsGenie is native, so no proxy is needed. Severity maps to priority P1, P3, P5. The dedup alias combines the rule id and the firing timestamp, so a later firing opens its own alert instead of folding into the first.

Generic webhook posts the alert payload as JSON: rule_id, rule_name, metric, operator, threshold, current_value, tenant_id, severity, fired_at. Datadog, or anything else that accepts a JSON POST, uses this type with channel_type: "webhook".

Dispatch failures are reported rather than swallowed: missing_url, blocked_url, missing_routing_key, missing_api_key, unsupported_channel, timeout, network_error, or http_ followed by the status code. Outbound dispatch aborts after ten seconds.

Use the test_channel action before you rely on a rule. It returns 200 {"status":"sent","channel_type":"..."} only when the receiver answered 2xx; a configuration problem answers 400 and a delivery failure 502, both with {"status":"failed","reason":"...","error":"..."}.

Every administrative and security-relevant action is appended to shield_audit_log through the shared writeAuditLog() helper. The table is the evidence base for SOC 2 readiness and ISO 27001 work.

The raw IP address is never stored. It is hashed with a keyed HMAC under the server-held pepper at the single choke point in the logger, because a plain digest of a 32-bit address space can be reversed by enumeration. A daily pg_cron job at 03:00 UTC (anonymize-audit-log-pii) then clears actor_ip_hash and actor_user_agent from rows older than 90 days. Those two columns are also revoked from the anon and authenticated roles, so a browser client cannot read them at all.

Tamper evidence. Every audit row, every signal event row and every entropy-pool seed record is linked at commit into an append-only hash chain (shield_chain_ledger, chains audit_log, events and entropy_pool): each entry binds the previous link, the row's position, its id and a digest of its content. A pg_cron job every five minutes signs the chain heads and anchors the checkpoint to the latest NIST Randomness Beacon pulse, so a checkpoint cannot have been formed before that public pulse existed, and re-verifies the ledger since the previous checkpoint. The two columns the 90-day scrub clears are excluded from the content hash by design; everything else on the row is covered. The latest checkpoint and its public key are public at GET https://api.passkeybridge.io/v1/shield-chain-checkpoint, and any earlier one with ?seq=N.

Reading it. Dashboard > Audit log lists the 200 most recent entries for the organization, with a text search over action, resource type and resource id, a filter by action, expandable rows showing changes, metadata and error_message, counts of success, failure and denied, and a CSV export whose cells starting with =, +, - or @ are prefixed with an apostrophe against spreadsheet formula injection.

When a playbook's webhook_callback action fires, the attempt is recorded in shield_webhook_deliveries.

retrying means the first attempt failed with a retryable status and the delivery was handed to the queue worker for backoff. A failure that is not worth retrying is recorded as failed immediately.

Reading it. Dashboard > Webhooks, in the Automation group, titled "Webhook Deliveries". It lists the 50 most recent attempts for the organization, newest first, each row expanding to show the request body, the response body and any error. It is not on the Observability tab. Signing and payload shape for these callbacks are covered in Webhook configuration.

Dashboard > Observability, in the Operations group. Available on every plan.

• Four tiles across the top for the last 24 hours: Events with the number of distinct types, P95 latency with the P50 beneath it, High risk with the mean risk score, and Agent trust with the count of active delegates. A tile reads "Not yet available" rather than zero when the window has no measurement.
• Decision latency, with 1 hour, 24 hours and 7 days buttons. It shows the share of timed signals inside the target, the mean, a distribution over seven fixed buckets, and, separately, the end-to-end figure including outbound actions. Below 20 timed signals it shows a range instead of percentiles and says why.
• Anomalies, with a window selector (6 hours, 24 hours, 3 days, 7 days) and a refresh control.
• Alert rules, a collapsed section holding the rule list: a switch to activate or deactivate each rule, a test control that sends a synthetic notification, and a delete control with confirmation. The inline create form offers Slack, PagerDuty and a custom webhook; the OpsGenie channel is available through the "New alert rule" wizard in the tab header, which walks metric, threshold and severity, channel, then review.
• Alert history, filtered by All, Pending or Acknowledged, each firing showing rule name, severity, channel, metric value and threshold, with an Acknowledge button.

Cards fetch on open and on refresh. There is no realtime subscription and no auto-refresh, so a card shows the window as it stood when you loaded it.