Skip to content

Router.Africa — Observability Plan

Backlog item #14. What signals tell us something is wrong before a customer does, which of them wake someone up, and what to do first. The tooling is decided (Datadog, decisions-log.md → Observability tooling). This document is the signals and alerts plan on top of it. Every threshold is proposed: launch traffic is assumed to be about 20 req/s or less, so tune them against the k6 load test and real traffic (section 10).


  1. Money correctness first. The worst failure is a wrong balance, not a slow page. Anything that can leave the ledger wrong pages; anything that only degrades a screen does not.
  2. Correctness is checked by the app, not inferred from graphs. The ledger invariant check, the usage reconciliation, the key reconciliation and the top-up status check already exist as scheduled jobs (backend-scope.md section 5). Datadog’s job is to make sure those jobs ran and to alert on what they find, not to re-derive it.
  3. Page rarely, and only for something a person must do now. Three tiers (section 4). Everything else is a ticket or a digest.
  4. Real-time limits never come from the async log. Rate limiting and caps are enforced in Bifrost and Postgres. The request log and rollups are written asynchronously, so they are good for detection and dashboards, never for enforcement.
  5. Never capture secrets or content. Prompts, responses, cookies and keys stay out of traces and logs (section 3).

One platform for the whole stack (decided). Nothing here adds a second vendor.

Component How it reports to Datadog Notes
Bifrost (K3s) Native datadog plugin: APM traces, agent or agentless. Bifrost’s telemetry (Prometheus) is a cheap extra for provider-level metrics Content logging stays off (disable_content_logging), so traces carry tokens, cost, status and latency, not prompts
apps/api, apps/dashboard-api (Workers) Workers native OpenTelemetry export, automatic instrumentation Custom spans around the hold, the Bifrost call, the settle and the queue publish, so latency is broken down by stage
apps/niobi-proxy (Node, K3s) dd-trace or OTel export Includes calls to Niobi and signature verification outcomes
K3s droplet Datadog Agent on the node (host CPU, memory, disk, pod restarts) The droplet is one machine, so host metrics matter more than they would on a cluster
Postgres (cluster) Pod and volume metrics, backup status (last WAL archived, last base backup) and Hyperdrive metrics; slow-query and connection data The watched capacity number is the Hyperdrive pool
Cloudflare Queues and R2 Queue backlog and age; R2 error counts
Frontends (apps/dashboard, apps/admin) Optional: real user monitoring and JS errors, with request bodies and stored content excluded Not needed at launch; add if support tickets need it
Scheduled jobs Datadog reads logs.job_runs and core.scheduled_jobs (or the dispatcher emits a metric per run) Section 5.6
Synthetic checks Datadog Synthetics against api., app., admin. and a canary request Section 6

Correlation. x-request-id is our request id: it’s on every API response, is sent to Bifrost, is the Bifrost log id, and is the request-log primary key. Put it on every span and log line so one id links the Worker trace, the Bifrost trace, the request-log row and a customer’s support ticket. Deploy markers are sent from CI so a change lines up with a graph (cicd.md section 6).

Extends the rules in auth-security.md 3.4 and the decisions log (Audit trail & prompt/response retention).

  • Cookie, Set-Cookie, Authorization, x-api-key and any raw API key: stripped from traces, Workers logs, error reports and queue messages. A lint rule forbids logging a whole request or headers object, and conformance test 20 fails the build if one appears.
  • Prompt and response content: never in Datadog traces, and never in Bifrost’s own logging.
  • API keys appear only as the masked prefix (sk-ra-…abcd).
  • Customer email addresses and free text in tags: use ids. High-cardinality tags (per key, per request id) go on traces and logs, not on metrics, to keep the bill under control.
  • Cost control: sampling rates and log volume are configurable, not hard-coded (future-optimizations register), so cutting signal never needs a code change in an incident.
Tier Meaning Who and how Response
P1 Page Money may be wrong, the API is down, or customers cannot pay On-call, immediately (phone or push) Start now, any hour
P2 Notify Degraded or drifting, will become P1 if ignored Team channel, working hours, with a named owner Same day
P3 Digest Worth knowing, no action needed today Daily summary Review

Alert destination (Slack, email, PagerDuty) and who is on call are not decided; section 10.

Threshold is the proposed trigger. First response is what the person does first, so each alert links to one.

5.1 API availability and latency (apps/api)

Section titled “5.1 API availability and latency (apps/api)”
Signal Source Threshold Tier First response
Error rate (5xx), split by cause: our fault vs a provider Worker metrics + Bifrost Ours: > 1% of requests for 5 min. Provider-caused: > 5% for 10 min P1 ours / P2 provider Check the last deploy marker; roll back (cicd.md section 7). For a provider, check the fallback chain and provider status
Availability probe Synthetic on api.router.africa 2 consecutive failures from 2 regions P1 Cloudflare status, Worker deploy, Hyperdrive
Canary request fails Section 6 Any failure, twice in a row P1 The stage that failed (auth, hold, Bifrost, settle) is in the trace
Latency added by us (Worker time excluding the Bifrost call) Custom spans p95 > 300 ms for 10 min (tune after load test) P2 Hyperdrive pool, Postgres latency
Time to first token Streaming spans p95 up more than 50% against the 7-day baseline P2 Provider or Bifrost, not us; tell the customer-facing status page if sustained
Postgres unreachable from the request path Worker errors on the hold write Any sustained failure P1 Requests are being refused by design (Bifrost is never called without a hold); check the Postgres pod, the droplet and Hyperdrive
409 on idempotency claims Request log Sudden rise against baseline P3 Likely a client retry storm or a stuck in_progress row
Signal Source Threshold Tier First response
Ledger invariant check finds a difference ledger_invariant_check job Any workspace P1 Do not adjust blindly. Identify the workspace; a stuck open hold is the usual cause (backend-scope section 6); otherwise treat as a hot-path bug and stop deploys
Stuck hold holds with status open Older than the longest expected request (e.g. 10 min) P2 Usage reconciliation will find the call; an admin charges or releases it with a reason
Unbilled usage (not_charged rows) usage_discrepancies Any row older than 1 hour, or > N rows in a day (N tuned) P2 Admin Unbilled usage screen; charge or release. A rising count means the Worker is dying between settle and response
token_mismatch rows usage_discrepancies Any P2 Provider partial-billing coverage (decisions log, Mid-stream client disconnect policy); investigate per provider
Absorbed shortfall settlements.absorbed_shortfall, usage_hourly Rate above baseline, or one workspace above a set amount P2 Tune input_allowance; check whether holds are off for that org
Unusual manual money movement Audit events ledger.adjusted, hold_toggle.changed Each event (info); many in a day P3 / P2 Review actor and reason
Usage reconciliation itself not running Job monitor, section 5.6 No successful run in 3 hours P1 Without it, undetected unbilled usage accumulates
Bifrost cost against charged revenue (margin) usage_hourly (gateway_cost_usd vs charged) Margin below a floor for a model P2 Rate job stale, markup or FX wrong
Signal Source Threshold Tier First response
Daily rates not published today daily_rates job / rates table Not published by an agreed time P1 Stale rates mean wrong prices. Check the FX source (Frankfurter) and Bifrost cost read; run the job from the admin Scheduled jobs screen
FX rate fetch failing Job run summary Any failure P2 Retry; check the source
A model is “on” with no current rate, or a rate published from a Bifrost cost of 0 Rates job Any P1 Turn the model off; usage on it cannot be priced (decisions log, v1 model list)
Model catalog sync failing Job status 2 in a row P3 New models just stay off, so nothing breaks
Signal Source Threshold Tier First response
Queue consumer backlog age Cloudflare Queues Oldest message > 10 min P2 Consumer errors; apps/dashboard-api deploy. Money is already safe in the ledger; only logs and rollups are late
Dead-letter or repeated retries Queue Any messages in the DLQ P2 Inspect the event; fix the consumer; replay (idempotent)
Queue publish failures from apps/api Worker logs > 0.1% for 10 min P2 Logged rows will be missing; confirm the Queues status
R2 content write failures Worker logs > 1% for 15 min P2 Never fails the request, but stored content is missing for those calls
Analytics writes starving the ledger Hyperdrive pool, Postgres lock waits Pool near its limit, or analytics queries above a duration P2 The trigger to move analytics to its own instance (future-optimizations)
Log table size Postgres Largest log table nears 100M rows or 50 GB P3 The trigger to partition (future-optimizations)
R2 storage growth content_storage_daily Growth ahead of the launch estimate P3 Backlog #17 storage estimate

5.5 The gateway (Bifrost), the droplet, the network and Niobi

Section titled “5.5 The gateway (Bifrost), the droplet, the network and Niobi”
Signal Source Threshold Tier First response
Bifrost /health failing Datadog agent in the cluster + k8s probe (Bifrost has no public hostname after lockdown; the canary covers the Workers’ path) 2 failures P1 Pod restarts, the Bifrost Postgres pod, disk
Tunnel (cloudflared) down or Workers VPC binding errors Cloudflare tunnel health, Worker errors on the binding Any tunnel down; binding errors > 1% for 2 min P1 Every request and every top-up fails: the request path cannot reach Bifrost. Check the cloudflared pods (run 2 replicas) and the droplet
Unexpected open port on the droplet External port scan (scheduled synthetic) Anything other than the WireGuard UDP port answering, or 443 answering a non-Niobi IP P2 Firewall drift: check the DigitalOcean Cloud Firewall and any new Service of type LoadBalancer or NodePort
Droplet CPU, memory, disk Agent CPU > 80% for 10 min; disk > 80% P2 One machine is the ceiling (tech-stack.md); Bifrost’s own Postgres grows with its logs
Pod restarts or crash loop Agent > 2 restarts in 30 min P2 Logs; the memory limit in values.yaml (GOMEMLIMIT)
TLS certificate expiry (Niobi webhook host; the Bifrost ingress until lockdown) cert-manager status < 14 days P2 cert-manager renewal. After lockdown port 80 is closed, so the webhook certificate uses the DNS-01 challenge (Cloudflare API token); the webhook DNS record stays un-proxied so the firewall sees Niobi’s IP
Bifrost Postgres backup missing Backup job No successful backup in 24 h P2 No backup job exists yet (installation-as-built.md, remaining work); this alert needs one first
Per-provider error rate or latency Bifrost telemetry Errors > 5% for 10 min P2 Fallback chain; provider status
Provider key rejected or out of quota Bifrost errors (401/402/403 from provider) Any P1 That provider’s models fail for everyone; rotate or top up the provider key
429 from Bifrost’s own rate limit Worker Trending up for one org P3 Expected behaviour for a customer at their limit; watch for a mis-set limit
niobi-proxy down Health call from dashboard-api through the VPC binding, or dashboard-api errors 2 failures P1 Customers can’t top up; check the pod and the tunnel
niobi-callbacks queue backlog or push failures Cloudflare Queues; niobi-proxy logs Oldest message > 5 min, or any push failure P2 Credits are late, not lost: the top-up status check still resolves them. Check the queue API token in niobi-proxy
Niobi call error rate niobi-proxy > 5% for 10 min P2 Niobi status; Niobi’s own IP allowlist
Inbound Niobi signature verification failures niobi-proxy Any burst P2 Misconfigured credentials, or someone probing the callback
Top-ups pending past the threshold topups (status pending) Any older than the threshold P2 Top-up status check; the admin Transactions view flags them; nothing auto-fails
Callback lag Time from top-up start to credited p95 above a set value P3 Falling back to the status check is working, but the callback path may be broken
Signal Source Threshold Tier First response
Self-hosted runner offline GitHub Actions Offline for 30 min P3 Droplet deploys are blocked (Workers still deploy); check the runner pod
Edge rate-limit rule firing on api. Cloudflare analytics Sustained blocks from many IPs P2 Likely an attack or a customer misconfiguration; compare with the per-key 429 trend
Edge rate-limit rule changed Audit log (rate_limit.changed) Any change P3 Confirm the change was intended

The dispatcher fires every minute; each job records a row in logs.job_runs. Alert on the absence of a good run, not only on failures.

Signal Threshold Tier First response
Any enabled job: no successful run within 2 × its schedule interval Per job Job-specific: P1 for daily_rates, usage_reconciliation, ledger_invariant_check, topup_status_check; P2 for the rest Admin Scheduled jobs screen: last error, then Run now
Job last_status = failed or partial 2 consecutive P2 Run history error text
Dispatcher itself not firing No job started in 5 min P1 The Cron Trigger, the dashboard-api deploy
A job disabled in the admin screen Any P3 The disable reason is audit-logged; confirm it is intended, and re-enable
Job lease reclaimed (a run died) Any P2 The run died mid-way; check the summary and re-run
Key discrepancies (gateway_discrepancies) Any older than 1 hour P2 Admin discrepancy view; re-sync or clean up
Keys stuck pending Any older than 10 min P2 Bifrost create failed after the row was written; the customer never saw a key
Low-balance alert emails not sent Job status P3 Resend status

Key-misuse detection is in api-key-security.md section 7; these are the platform-level signals.

Signal Source Threshold Tier First response
401 invalid_api_key flood Worker metrics Above baseline, especially many distinct keys from few sources P2 Guessing or stuffing; the checksum already rejects malformed keys before the database. Add a Cloudflare rule
Auth endpoint abuse Sign-in failures, Turnstile failures, rate-limit hits Burst P2 Tighten limits; check for a credential-stuffing run
Staff sign-in outside expected hours or from a new device Admin session events Any (info) P3 Confirm with the person
Stored-content reads Audit event content.read Volume above baseline for one actor P2 Who and why; step-up already required
Impersonation started impersonation.started Each (info) P3 Review reason
Sessions killed en masse Deletes on sessions Any bulk delete P2 Confirm it is the breach kill switch or a deploy, not an incident
Cloudflare cache rule or header regression on app. / admin. Header test in CI plus a synthetic header check Cache-Control: no-store missing P1 One user could be served another’s response (auth-security.md 3.3)
Email deliverability Resend bounces and complaints Bounce rate above a set level P3 Verification and reset emails not arriving
  • Probes: api.router.africa health and app.router.africa load, from more than one region, every minute. admin.router.africa sits behind Cloudflare Access, so its probe uses an Access service token. Bifrost /health is checked from inside the cluster, since Bifrost has no public hostname after lockdown.
  • End-to-end canary (proposed): every 5 minutes, one tiny real request through the full path (key auth, model gate, hold, Bifrost, settle, queue) using a dedicated internal workspace and key. It costs a few credits a day, and it is the only signal that proves the whole money path works, not just that a process is up. The internal workspace is excluded from customer-facing rollups, and its key follows the same rules as any key.
  • Status page (decided for launch): the public status page (../product/api-surface.md) is fed by the availability probe and the canary, so customers see the same signal we do. Tooling and location are still to choose.

Four, each answering one question.

Dashboard Question Contents
Request path Is the API healthy? Requests, error rate by cause, latency by stage (auth, hold, Bifrost, settle), idempotency replays, streaming disconnects
Money Is the ledger right? Invariant check result, open holds and their age, unbilled usage, absorbed shortfall, rate freshness, margin per model, top-ups pending
Platform Will the platform hold? Hyperdrive pool, Postgres latency and backup age, queue backlog and age, droplet host metrics, pod restarts, certificate expiry
Jobs and sync Are the background jobs alive? Each job’s last success and next run, dispatcher heartbeat, key discrepancies

Customer-facing usage and spend views are a product feature backed by usage_hourly, not a Datadog dashboard.

Set once there is a load test and a month of traffic. Starting points, so alerts have something to measure against:

Objective Target
apps/api availability, counting only failures that are ours (not a provider’s) 99.9% monthly
Latency we add on top of the provider (p95) Under 300 ms
Ledger invariant clean 100% of daily checks
Rates published before the start of the business day Every day
Unbilled usage found and resolved Within 1 business day
Top-ups resolved (credited or failed) Within 15 minutes of the customer paying

Each P1 and P2 alert names its first response above; these are the longer procedures that don’t exist yet.

  1. Ledger invariant failure.
  2. A stuck hold or unbilled usage (charge or release, with a reason).
  3. Rates job failed or stale.
  4. Provider outage and fallback.
  5. Queue backlog and replay.
  6. Niobi or Tunnel outage (top-ups down, customer messaging).
  7. Breach: kill switch for sessions, secret rotation (auth-security.md 3.2).
  8. Leaked API key report (api-key-security.md).
  9. Droplet full or unhealthy; restore from backup.
  • Alert destination and on-call: where P1 pages go (Slack, PagerDuty, phone) and who is on call. A small team needs an explicit rota before launch.
  • Thresholds: every number above; set from the k6 load test and the first weeks of traffic.
  • Datadog spend: the plan and log volume assumptions. The register already notes it is pricier than Sentry at scale; the sampling settings are the lever.
  • Bifrost’s alerting plugin (declarative Slack, PagerDuty and webhook rules, plugins-and-extensibility.md) as a second path for gateway-side alerts, versus routing everything through Datadog. Datadog alone is simpler and matches the decision to have one platform.
  • How job results reach Datadog: reading logs.job_runs, or the dispatcher emitting a metric per run. The second is simpler to alert on.
  • A missing backup job for Bifrost’s Postgres (installation-as-built.md) blocks the backup alert in 5.5.
  • Whether to store the client IP or country on the request log (this affects key-misuse detection; see api-key-security.md section 7).