Skip to content

Phase 9 — Pre-Launch Hardening

Goal: make the system safe to open:

  • the droplet lockdown, and every amber item on the network map closed or consciously accepted
  • observability with real alerts, the canary and the status page
  • a load test that sets the production database tier
  • legal text and developer docs
  • a production environment verified end to end by staff, before any customer

Milestone M3, launch-ready:

  • every task below is done
  • the network map has no red items and its amber items are resolved or accepted in writing
  • a staff workspace in production has topped up with real money through Niobi and made a real, correctly charged request

Ordering rule: P9.01 (lockdown) must be done before real provider keys carry real traffic in production (P9.14).

ID Task Who Needs Status
P9.01 Droplet lockdown You + agent P0.10, P0.09 todo
P9.02 Niobi webhook path in production You + agent P9.01, P6.03 todo
P9.03 Kubernetes network policies Agent P0.10, P6.01 todo
P9.04 Cloudflare Access in production You P8.01 todo
P9.05 Edge rules: HTTP refusal and production rate limits You + agent P8.07 todo
P9.06 Bifrost: content logging, backups, pinned image You + agent P0.08 todo
P9.07 Observability: telemetry, scrubbing, alerts, dashboards You + agent Phases 3–6 todo
P9.08 Canary, synthetics and status page You + agent P9.07, P3.11 todo
P9.09 Runbooks Agent P9.07 todo
P9.10 Load test and the production database tier You + agent Phases 3–6 todo
P9.11 Security review and open policies You + agent P9.01–P9.06 todo
P9.12 Terms, privacy and storage estimate You + agent P5.06 todo
P9.13 Developer docs live (Track D gate) You + agent D.09 todo
P9.14 Production build-out and staff verification (M3) You + agent All above todo

Who: You + agent · Needs: P0.10, P0.09

In order:

  1. Set up WireGuard: the droplet peer and your laptop peer, with keys in Bitwarden.
  2. Confirm SSH, kubectl and the Bifrost dashboard work over it.
  3. Test the DigitalOcean web console as the recovery path.
  4. Create the DigitalOcean Cloud Firewall: inbound only the WireGuard UDP port, outbound open. The Niobi rule comes in P9.02.
  5. Close public SSH, 80 and 443. Stop relying on ufw.
  6. Remove the prod-bifrost-core.router.africa ingress and DNS record.
  7. Add the scheduled external port scan.

P9.02 — Niobi webhook path in production

Section titled “P9.02 — Niobi webhook path in production”

Who: You + agent · Needs: P9.01, P6.03

  • You:

    • pick the webhook hostname
    • add a DNS-only (not proxied) record
    • give Niobi the callback URL
    • add the firewall rule allowing 443 only from Niobi’s source IPs
  • Agent:

    • cert-manager with a DNS-01 issuer, using the DNS-edit token
    • a Traefik route that forwards only the webhook path to niobi-proxy
  • Spec: decisions log, Niobi webhooks; ../architecture/network-security-map.md (A6)

  • Done when:

    • a request from a non-Niobi IP gets no answer
    • a Niobi test callback reaches the queue
    • the certificate renews by DNS-01

Who: Agent · Needs: P0.10, P6.01

Default-deny in every namespace. Allow only:

  • cloudflared → the gateway proxy and niobi-proxy

  • Traefik → niobi-proxy (webhook)

  • the gateway proxy → Bifrost

  • Bifrost → its Postgres

  • cloudflared → the staging Postgres (Hyperdrive’s path), and the runner → the staging Postgres (migrations)

  • each pod’s required egress

  • Spec: decisions log, Kubernetes network policies

  • Done when: from a niobi-proxy pod, Bifrost’s Postgres is unreachable, and every allowed path still works (staging and production smoke pass).

Who: You · Needs: P8.01

Put admin.router.africa behind Access: a named allow-list (you) with the email one-time PIN. Create an Access service token for the Datadog admin probe.

  • Spec: decisions log, Admin origin access
  • Done when: an unlisted email can’t load the admin app, and the probe passes with its token.

P9.05 — Edge rules: HTTP refusal and production rate limits

Section titled “P9.05 — Edge rules: HTTP refusal and production rate limits”

Who: You + agent · Needs: P8.07

  • HTTP refusal: choose and apply the Cloudflare rule that refuses plain HTTP on api.router.africa instead of redirecting it, and record the choice.

  • Rate limits: set the production rate-limit values in the admin app.

  • Spec: ../architecture/api-key-security.md sections 6 and 10; decisions log, Edge rate limiting

  • Done when: curl http://api.router.africa/v1/models is refused without a redirect, and the production rules show in admin screen 55.

P9.06 — Bifrost: content logging, backups, pinned image

Section titled “P9.06 — Bifrost: content logging, backups, pinned image”

Who: You + agent · Needs: P0.08

  • Content logging: confirm disable_content_logging is set, and verify no prompt content lands in Bifrost’s Postgres.

  • Backups: a pg_dump CronJob to off-cluster storage (e.g. DigitalOcean Spaces), plus one restore test.

  • Image: confirm the pinned image is recorded and a copy is kept.

  • Spec: backlog #17; ../infra/installation-as-built.md remaining work, item 3; decisions log, Gateway abstraction (pinning)

  • Done when: a test prompt leaves no content in Bifrost’s database, a backup from the last 24 hours exists, and a restore into a scratch namespace works.

P9.07 — Observability: telemetry, scrubbing, alerts, dashboards

Section titled “P9.07 — Observability: telemetry, scrubbing, alerts, dashboards”

Who: You + agent · Needs: phases 3–6

  • Telemetry: Datadog through the Workers OpenTelemetry export, plus the droplet agent.

  • Spans: custom spans for auth, hold, the gateway call, settle and the queue publish.

  • Scrubbing: Authorization, cookies, keys and prompt content are scrubbed, and that is verified.

  • Alerts: every signal in observability.md section 5, with its tier.

  • Dashboards: the dashboards in section 7.

  • Deploy markers: sent from the pipeline.

  • On-call routing: decide it (it isn’t decided yet) and record it.

  • Spec: ../architecture/observability.md; backlog #17 Datadog

  • Done when: a search in Datadog for a test prompt and a test key finds nothing, and each P1 alert has fired once in a staging drill.

P9.08 — Canary, synthetics and status page

Section titled “P9.08 — Canary, synthetics and status page”

Who: You + agent · Needs: P9.07, P3.11

  • Canary: the P3.11 request every 5 minutes in production, on an internal workspace.

  • Synthetics: api. and app. probes from several regions, and the admin probe through Access.

  • Status page: decide the tool, then publish the status page, fed by the probes and the canary. Left open by the operator on 2026-11-07, to decide here. Standing proposal: a hosted status-page service on a status. subdomain, not something we run ourselves. Reasons: it must stay up when our stack is down (so not on the 4 GB droplet, and not on the same Workers as api.), Datadog already runs the probes and can notify it by webhook, and it adds no server to operate. Shortlist to compare on price, custom domain on the free or low tier, webhook or API updates, and subscriber emails: Instatus, Better Stack, Atlassian Statuspage. A hand-built static page on Cloudflare Pages is the cheapest fallback but means updating it ourselves. The page is public, so it lists only api. and app., never admin.. Adding status. is an extra hostname outside the four-origin split; record it in the decisions log when chosen.

  • Spec: ../architecture/observability.md section 6; ../product/api-surface.md section 5

  • Done when: the status page is live and reflects a forced canary failure in staging.

Who: Agent · Needs: P9.07

Write the nine runbooks listed in observability.md section 9, plus the leaked-key runbook in api-key-security.md 7.3. Each one covers: how to detect it, first steps, the commands, and how to communicate with customers.

P9.10 — Load test and the production database tier

Section titled “P9.10 — Load test and the production database tier”

Who: You + agent · Needs: phases 3–6

k6 against the production database (the cluster Postgres) before any customer is on it, at the stated peak (about 1,000 writes per second), with hold/settle on one hot workspace and spread across many. Watch the Hyperdrive pool, Postgres latency, the droplet’s CPU and memory (Bifrost shares it) and the added Worker latency.

  • Decide: whether the cluster Postgres holds the launch load or production moves to PlanetScale before launch (decisions log, Ledger database), and the Hyperdrive pool size.

  • Tune: the alert thresholds that were waiting on load-test data.

  • Spec: decisions log, Ledger database; ../architecture/observability.md (thresholds to tune); ../architecture/future-optimizations.md Ledger tier upgrade

  • Done when: the database decision is recorded, and the p95 added latency is within the observability target at peak.

P9.11 — Security review and open policies

Section titled “P9.11 — Security review and open policies”

Who: You + agent · Needs: P9.01–P9.06

  • Re-run the checks: both conformance suites, the header test against production hostnames, and the secret scan over the full history.

  • Walk the network map: every amber item is either closed or accepted in writing.

  • Decide the policy for credits spent through a leaked key, and the remaining items in api-key-security.md section 10.

  • Workers VPC: check its pricing and status.

  • Secrets: confirm the inventory is complete.

  • Spec: ../architecture/network-security-map.md; ../architecture/api-key-security.md section 10; ../infra/secrets-inventory.md

  • Done when: the network map is updated, the policies are recorded in the decisions log, and every secret in use has an inventory row.

P9.12 — Terms, privacy and storage estimate

Section titled “P9.12 — Terms, privacy and storage estimate”

Who: You + agent · Needs: P5.06

  • Website repo: terms of service and privacy policy that say plainly that prompts and responses are not saved unless the customer opts in, the retention of what is saved, and who can read it.

  • Storage: estimate R2 growth against expected traffic, given the indefinite retention.

  • Spec: backlog #17 Website, Storage estimate; decisions log, Audit trail & prompt/response retention

  • Done when: both documents are published on router.africa, sign-up links to them, and the storage estimate is recorded in backlog #17.

P9.13 — Developer docs live (Track D gate)

Section titled “P9.13 — Developer docs live (Track D gate)”

Who: You + agent · Needs: D.09

The developer docs are built in their own track, track-d-developer-docs.md: a custom docs app, the curated OpenAPI spec, the Request Builder and the AI-readable outputs. This task is done when that track’s launch gate, D.09, is done. D.10 (Ask AI) is not required.

P9.14 — Production build-out and staff verification (M3)

Section titled “P9.14 — Production build-out and staff verification (M3)”

Who: You + agent · Needs: all above

  1. You: add production provider keys to Bifrost, over WireGuard.
  2. You: set every production secret per the inventory.
  3. You + agent: restore drill. Restore the production database from R2 to a point in time into a scratch namespace, and check the ledger matches. If P9.10 chose PlanetScale, do the move here instead, then run the same check against it.
  4. Cut release/0.9 and run Deploy to production for v0.9.0.
  5. In a staff-only workspace:
    • top up a small real amount through Niobi
    • make real requests
    • check the charge, the ledger, the reconciliations (both clean), the margin report and the alerts