Router.Africa — Observability Plan
Backlog item #14. What signals tell us something is wrong before a customer does, which of them wake someone up, and what to do first. The tooling is decided (Datadog, decisions-log.md → Observability tooling). This document is the signals and alerts plan on top of it. Every threshold is proposed: launch traffic is assumed to be about 20 req/s or less, so tune them against the k6 load test and real traffic (section 10).
1. Principles
Section titled “1. Principles”- Money correctness first. The worst failure is a wrong balance, not a slow page. Anything that can leave the ledger wrong pages; anything that only degrades a screen does not.
- Correctness is checked by the app, not inferred from graphs. The ledger invariant check, the usage reconciliation, the key reconciliation and the top-up status check already exist as scheduled jobs (
backend-scope.mdsection 5). Datadog’s job is to make sure those jobs ran and to alert on what they find, not to re-derive it. - Page rarely, and only for something a person must do now. Three tiers (section 4). Everything else is a ticket or a digest.
- Real-time limits never come from the async log. Rate limiting and caps are enforced in Bifrost and Postgres. The request log and rollups are written asynchronously, so they are good for detection and dashboards, never for enforcement.
- Never capture secrets or content. Prompts, responses, cookies and keys stay out of traces and logs (section 3).
2. What is instrumented, and how
Section titled “2. What is instrumented, and how”One platform for the whole stack (decided). Nothing here adds a second vendor.
| Component | How it reports to Datadog | Notes |
|---|---|---|
| Bifrost (K3s) | Native datadog plugin: APM traces, agent or agentless. Bifrost’s telemetry (Prometheus) is a cheap extra for provider-level metrics |
Content logging stays off (disable_content_logging), so traces carry tokens, cost, status and latency, not prompts |
apps/api, apps/dashboard-api (Workers) |
Workers native OpenTelemetry export, automatic instrumentation | Custom spans around the hold, the Bifrost call, the settle and the queue publish, so latency is broken down by stage |
apps/niobi-proxy (Node, K3s) |
dd-trace or OTel export |
Includes calls to Niobi and signature verification outcomes |
| K3s droplet | Datadog Agent on the node (host CPU, memory, disk, pod restarts) | The droplet is one machine, so host metrics matter more than they would on a cluster |
| Postgres (cluster) | Pod and volume metrics, backup status (last WAL archived, last base backup) and Hyperdrive metrics; slow-query and connection data | The watched capacity number is the Hyperdrive pool |
| Cloudflare Queues and R2 | Queue backlog and age; R2 error counts | |
Frontends (apps/dashboard, apps/admin) |
Optional: real user monitoring and JS errors, with request bodies and stored content excluded | Not needed at launch; add if support tickets need it |
| Scheduled jobs | Datadog reads logs.job_runs and core.scheduled_jobs (or the dispatcher emits a metric per run) |
Section 5.6 |
| Synthetic checks | Datadog Synthetics against api., app., admin. and a canary request |
Section 6 |
Correlation. x-request-id is our request id: it’s on every API response, is sent to Bifrost, is the Bifrost log id, and is the request-log primary key. Put it on every span and log line so one id links the Worker trace, the Bifrost trace, the request-log row and a customer’s support ticket. Deploy markers are sent from CI so a change lines up with a graph (cicd.md section 6).
3. What must never appear in telemetry
Section titled “3. What must never appear in telemetry”Extends the rules in auth-security.md 3.4 and the decisions log (Audit trail & prompt/response retention).
Cookie,Set-Cookie,Authorization,x-api-keyand any raw API key: stripped from traces, Workers logs, error reports and queue messages. A lint rule forbids logging a whole request or headers object, and conformance test 20 fails the build if one appears.- Prompt and response content: never in Datadog traces, and never in Bifrost’s own logging.
- API keys appear only as the masked prefix (
sk-ra-…abcd). - Customer email addresses and free text in tags: use ids. High-cardinality tags (per key, per request id) go on traces and logs, not on metrics, to keep the bill under control.
- Cost control: sampling rates and log volume are configurable, not hard-coded (future-optimizations register), so cutting signal never needs a code change in an incident.
4. Severity and routing
Section titled “4. Severity and routing”| Tier | Meaning | Who and how | Response |
|---|---|---|---|
| P1 Page | Money may be wrong, the API is down, or customers cannot pay | On-call, immediately (phone or push) | Start now, any hour |
| P2 Notify | Degraded or drifting, will become P1 if ignored | Team channel, working hours, with a named owner | Same day |
| P3 Digest | Worth knowing, no action needed today | Daily summary | Review |
Alert destination (Slack, email, PagerDuty) and who is on call are not decided; section 10.
5. Signals and alerts
Section titled “5. Signals and alerts”Threshold is the proposed trigger. First response is what the person does first, so each alert links to one.
5.1 API availability and latency (apps/api)
Section titled “5.1 API availability and latency (apps/api)”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
| Error rate (5xx), split by cause: our fault vs a provider | Worker metrics + Bifrost | Ours: > 1% of requests for 5 min. Provider-caused: > 5% for 10 min | P1 ours / P2 provider | Check the last deploy marker; roll back (cicd.md section 7). For a provider, check the fallback chain and provider status |
| Availability probe | Synthetic on api.router.africa |
2 consecutive failures from 2 regions | P1 | Cloudflare status, Worker deploy, Hyperdrive |
| Canary request fails | Section 6 | Any failure, twice in a row | P1 | The stage that failed (auth, hold, Bifrost, settle) is in the trace |
| Latency added by us (Worker time excluding the Bifrost call) | Custom spans | p95 > 300 ms for 10 min (tune after load test) | P2 | Hyperdrive pool, Postgres latency |
| Time to first token | Streaming spans | p95 up more than 50% against the 7-day baseline | P2 | Provider or Bifrost, not us; tell the customer-facing status page if sustained |
| Postgres unreachable from the request path | Worker errors on the hold write | Any sustained failure | P1 | Requests are being refused by design (Bifrost is never called without a hold); check the Postgres pod, the droplet and Hyperdrive |
409 on idempotency claims |
Request log | Sudden rise against baseline | P3 | Likely a client retry storm or a stuck in_progress row |
5.2 Ledger and billing correctness
Section titled “5.2 Ledger and billing correctness”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
| Ledger invariant check finds a difference | ledger_invariant_check job |
Any workspace | P1 | Do not adjust blindly. Identify the workspace; a stuck open hold is the usual cause (backend-scope section 6); otherwise treat as a hot-path bug and stop deploys |
| Stuck hold | holds with status open |
Older than the longest expected request (e.g. 10 min) | P2 | Usage reconciliation will find the call; an admin charges or releases it with a reason |
Unbilled usage (not_charged rows) |
usage_discrepancies |
Any row older than 1 hour, or > N rows in a day (N tuned) | P2 | Admin Unbilled usage screen; charge or release. A rising count means the Worker is dying between settle and response |
token_mismatch rows |
usage_discrepancies |
Any | P2 | Provider partial-billing coverage (decisions log, Mid-stream client disconnect policy); investigate per provider |
| Absorbed shortfall | settlements.absorbed_shortfall, usage_hourly |
Rate above baseline, or one workspace above a set amount | P2 | Tune input_allowance; check whether holds are off for that org |
| Unusual manual money movement | Audit events ledger.adjusted, hold_toggle.changed |
Each event (info); many in a day | P3 / P2 | Review actor and reason |
| Usage reconciliation itself not running | Job monitor, section 5.6 | No successful run in 3 hours | P1 | Without it, undetected unbilled usage accumulates |
| Bifrost cost against charged revenue (margin) | usage_hourly (gateway_cost_usd vs charged) |
Margin below a floor for a model | P2 | Rate job stale, markup or FX wrong |
5.3 Pricing inputs
Section titled “5.3 Pricing inputs”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
| Daily rates not published today | daily_rates job / rates table |
Not published by an agreed time | P1 | Stale rates mean wrong prices. Check the FX source (Frankfurter) and Bifrost cost read; run the job from the admin Scheduled jobs screen |
| FX rate fetch failing | Job run summary | Any failure | P2 | Retry; check the source |
| A model is “on” with no current rate, or a rate published from a Bifrost cost of 0 | Rates job | Any | P1 | Turn the model off; usage on it cannot be priced (decisions log, v1 model list) |
| Model catalog sync failing | Job status | 2 in a row | P3 | New models just stay off, so nothing breaks |
5.4 Queue, logging and storage
Section titled “5.4 Queue, logging and storage”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
| Queue consumer backlog age | Cloudflare Queues | Oldest message > 10 min | P2 | Consumer errors; apps/dashboard-api deploy. Money is already safe in the ledger; only logs and rollups are late |
| Dead-letter or repeated retries | Queue | Any messages in the DLQ | P2 | Inspect the event; fix the consumer; replay (idempotent) |
Queue publish failures from apps/api |
Worker logs | > 0.1% for 10 min | P2 | Logged rows will be missing; confirm the Queues status |
| R2 content write failures | Worker logs | > 1% for 15 min | P2 | Never fails the request, but stored content is missing for those calls |
| Analytics writes starving the ledger | Hyperdrive pool, Postgres lock waits | Pool near its limit, or analytics queries above a duration | P2 | The trigger to move analytics to its own instance (future-optimizations) |
| Log table size | Postgres | Largest log table nears 100M rows or 50 GB | P3 | The trigger to partition (future-optimizations) |
| R2 storage growth | content_storage_daily |
Growth ahead of the launch estimate | P3 | Backlog #17 storage estimate |
5.5 The gateway (Bifrost), the droplet, the network and Niobi
Section titled “5.5 The gateway (Bifrost), the droplet, the network and Niobi”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
Bifrost /health failing |
Datadog agent in the cluster + k8s probe (Bifrost has no public hostname after lockdown; the canary covers the Workers’ path) | 2 failures | P1 | Pod restarts, the Bifrost Postgres pod, disk |
Tunnel (cloudflared) down or Workers VPC binding errors |
Cloudflare tunnel health, Worker errors on the binding | Any tunnel down; binding errors > 1% for 2 min | P1 | Every request and every top-up fails: the request path cannot reach Bifrost. Check the cloudflared pods (run 2 replicas) and the droplet |
| Unexpected open port on the droplet | External port scan (scheduled synthetic) | Anything other than the WireGuard UDP port answering, or 443 answering a non-Niobi IP | P2 | Firewall drift: check the DigitalOcean Cloud Firewall and any new Service of type LoadBalancer or NodePort |
| Droplet CPU, memory, disk | Agent | CPU > 80% for 10 min; disk > 80% | P2 | One machine is the ceiling (tech-stack.md); Bifrost’s own Postgres grows with its logs |
| Pod restarts or crash loop | Agent | > 2 restarts in 30 min | P2 | Logs; the memory limit in values.yaml (GOMEMLIMIT) |
| TLS certificate expiry (Niobi webhook host; the Bifrost ingress until lockdown) | cert-manager status | < 14 days | P2 | cert-manager renewal. After lockdown port 80 is closed, so the webhook certificate uses the DNS-01 challenge (Cloudflare API token); the webhook DNS record stays un-proxied so the firewall sees Niobi’s IP |
| Bifrost Postgres backup missing | Backup job | No successful backup in 24 h | P2 | No backup job exists yet (installation-as-built.md, remaining work); this alert needs one first |
| Per-provider error rate or latency | Bifrost telemetry | Errors > 5% for 10 min | P2 | Fallback chain; provider status |
| Provider key rejected or out of quota | Bifrost errors (401/402/403 from provider) | Any | P1 | That provider’s models fail for everyone; rotate or top up the provider key |
429 from Bifrost’s own rate limit |
Worker | Trending up for one org | P3 | Expected behaviour for a customer at their limit; watch for a mis-set limit |
niobi-proxy down |
Health call from dashboard-api through the VPC binding, or dashboard-api errors |
2 failures | P1 | Customers can’t top up; check the pod and the tunnel |
niobi-callbacks queue backlog or push failures |
Cloudflare Queues; niobi-proxy logs |
Oldest message > 5 min, or any push failure | P2 | Credits are late, not lost: the top-up status check still resolves them. Check the queue API token in niobi-proxy |
| Niobi call error rate | niobi-proxy |
> 5% for 10 min | P2 | Niobi status; Niobi’s own IP allowlist |
| Inbound Niobi signature verification failures | niobi-proxy |
Any burst | P2 | Misconfigured credentials, or someone probing the callback |
| Top-ups pending past the threshold | topups (status pending) |
Any older than the threshold | P2 | Top-up status check; the admin Transactions view flags them; nothing auto-fails |
| Callback lag | Time from top-up start to credited | p95 above a set value | P3 | Falling back to the status check is working, but the callback path may be broken |
5.5a Delivery and edge controls
Section titled “5.5a Delivery and edge controls”| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
| Self-hosted runner offline | GitHub Actions | Offline for 30 min | P3 | Droplet deploys are blocked (Workers still deploy); check the runner pod |
Edge rate-limit rule firing on api. |
Cloudflare analytics | Sustained blocks from many IPs | P2 | Likely an attack or a customer misconfiguration; compare with the per-key 429 trend |
| Edge rate-limit rule changed | Audit log (rate_limit.changed) |
Any change | P3 | Confirm the change was intended |
5.6 Scheduled jobs and key sync
Section titled “5.6 Scheduled jobs and key sync”The dispatcher fires every minute; each job records a row in logs.job_runs. Alert on the absence of a good run, not only on failures.
| Signal | Threshold | Tier | First response |
|---|---|---|---|
| Any enabled job: no successful run within 2 × its schedule interval | Per job | Job-specific: P1 for daily_rates, usage_reconciliation, ledger_invariant_check, topup_status_check; P2 for the rest |
Admin Scheduled jobs screen: last error, then Run now |
Job last_status = failed or partial |
2 consecutive | P2 | Run history error text |
| Dispatcher itself not firing | No job started in 5 min | P1 | The Cron Trigger, the dashboard-api deploy |
| A job disabled in the admin screen | Any | P3 | The disable reason is audit-logged; confirm it is intended, and re-enable |
| Job lease reclaimed (a run died) | Any | P2 | The run died mid-way; check the summary and re-run |
Key discrepancies (gateway_discrepancies) |
Any older than 1 hour | P2 | Admin discrepancy view; re-sync or clean up |
Keys stuck pending |
Any older than 10 min | P2 | Bifrost create failed after the row was written; the customer never saw a key |
| Low-balance alert emails not sent | Job status | P3 | Resend status |
5.7 Security and abuse
Section titled “5.7 Security and abuse”Key-misuse detection is in api-key-security.md section 7; these are the platform-level signals.
| Signal | Source | Threshold | Tier | First response |
|---|---|---|---|---|
401 invalid_api_key flood |
Worker metrics | Above baseline, especially many distinct keys from few sources | P2 | Guessing or stuffing; the checksum already rejects malformed keys before the database. Add a Cloudflare rule |
| Auth endpoint abuse | Sign-in failures, Turnstile failures, rate-limit hits | Burst | P2 | Tighten limits; check for a credential-stuffing run |
| Staff sign-in outside expected hours or from a new device | Admin session events | Any (info) | P3 | Confirm with the person |
| Stored-content reads | Audit event content.read |
Volume above baseline for one actor | P2 | Who and why; step-up already required |
| Impersonation started | impersonation.started |
Each (info) | P3 | Review reason |
| Sessions killed en masse | Deletes on sessions |
Any bulk delete | P2 | Confirm it is the breach kill switch or a deploy, not an incident |
Cloudflare cache rule or header regression on app. / admin. |
Header test in CI plus a synthetic header check | Cache-Control: no-store missing |
P1 | One user could be served another’s response (auth-security.md 3.3) |
| Email deliverability | Resend bounces and complaints | Bounce rate above a set level | P3 | Verification and reset emails not arriving |
6. Synthetic checks and the canary
Section titled “6. Synthetic checks and the canary”- Probes:
api.router.africahealth andapp.router.africaload, from more than one region, every minute.admin.router.africasits behind Cloudflare Access, so its probe uses an Access service token. Bifrost/healthis checked from inside the cluster, since Bifrost has no public hostname after lockdown. - End-to-end canary (proposed): every 5 minutes, one tiny real request through the full path (key auth, model gate, hold, Bifrost, settle, queue) using a dedicated internal workspace and key. It costs a few credits a day, and it is the only signal that proves the whole money path works, not just that a process is up. The internal workspace is excluded from customer-facing rollups, and its key follows the same rules as any key.
- Status page (decided for launch): the public status page (
../product/api-surface.md) is fed by the availability probe and the canary, so customers see the same signal we do. Tooling and location are still to choose.
7. Dashboards
Section titled “7. Dashboards”Four, each answering one question.
| Dashboard | Question | Contents |
|---|---|---|
| Request path | Is the API healthy? | Requests, error rate by cause, latency by stage (auth, hold, Bifrost, settle), idempotency replays, streaming disconnects |
| Money | Is the ledger right? | Invariant check result, open holds and their age, unbilled usage, absorbed shortfall, rate freshness, margin per model, top-ups pending |
| Platform | Will the platform hold? | Hyperdrive pool, Postgres latency and backup age, queue backlog and age, droplet host metrics, pod restarts, certificate expiry |
| Jobs and sync | Are the background jobs alive? | Each job’s last success and next run, dispatcher heartbeat, key discrepancies |
Customer-facing usage and spend views are a product feature backed by usage_hourly, not a Datadog dashboard.
8. Objectives (proposed, to tune)
Section titled “8. Objectives (proposed, to tune)”Set once there is a load test and a month of traffic. Starting points, so alerts have something to measure against:
| Objective | Target |
|---|---|
apps/api availability, counting only failures that are ours (not a provider’s) |
99.9% monthly |
| Latency we add on top of the provider (p95) | Under 300 ms |
| Ledger invariant clean | 100% of daily checks |
| Rates published before the start of the business day | Every day |
| Unbilled usage found and resolved | Within 1 business day |
| Top-ups resolved (credited or failed) | Within 15 minutes of the customer paying |
9. Runbooks to write at build
Section titled “9. Runbooks to write at build”Each P1 and P2 alert names its first response above; these are the longer procedures that don’t exist yet.
- Ledger invariant failure.
- A stuck hold or unbilled usage (charge or release, with a reason).
- Rates job failed or stale.
- Provider outage and fallback.
- Queue backlog and replay.
- Niobi or Tunnel outage (top-ups down, customer messaging).
- Breach: kill switch for sessions, secret rotation (
auth-security.md3.2). - Leaked API key report (
api-key-security.md). - Droplet full or unhealthy; restore from backup.
10. To confirm
Section titled “10. To confirm”- Alert destination and on-call: where P1 pages go (Slack, PagerDuty, phone) and who is on call. A small team needs an explicit rota before launch.
- Thresholds: every number above; set from the k6 load test and the first weeks of traffic.
- Datadog spend: the plan and log volume assumptions. The register already notes it is pricier than Sentry at scale; the sampling settings are the lever.
- Bifrost’s
alertingplugin (declarative Slack, PagerDuty and webhook rules,plugins-and-extensibility.md) as a second path for gateway-side alerts, versus routing everything through Datadog. Datadog alone is simpler and matches the decision to have one platform. - How job results reach Datadog: reading
logs.job_runs, or the dispatcher emitting a metric per run. The second is simpler to alert on. - A missing backup job for Bifrost’s Postgres (
installation-as-built.md) blocks the backup alert in 5.5. - Whether to store the client IP or country on the request log (this affects key-misuse detection; see
api-key-security.mdsection 7).