Phase 9 — Pre-Launch Hardening
Goal: make the system safe to open:
- the droplet lockdown, and every amber item on the network map closed or consciously accepted
- observability with real alerts, the canary and the status page
- a load test that sets the production database tier
- legal text and developer docs
- a production environment verified end to end by staff, before any customer
Milestone M3, launch-ready:
- every task below is done
- the network map has no red items and its amber items are resolved or accepted in writing
- a staff workspace in production has topped up with real money through Niobi and made a real, correctly charged request
Ordering rule: P9.01 (lockdown) must be done before real provider keys carry real traffic in production (P9.14).
| ID | Task | Who | Needs | Status |
|---|---|---|---|---|
| P9.01 | Droplet lockdown | You + agent | P0.10, P0.09 | todo |
| P9.02 | Niobi webhook path in production | You + agent | P9.01, P6.03 | todo |
| P9.03 | Kubernetes network policies | Agent | P0.10, P6.01 | todo |
| P9.04 | Cloudflare Access in production | You | P8.01 | todo |
| P9.05 | Edge rules: HTTP refusal and production rate limits | You + agent | P8.07 | todo |
| P9.06 | Bifrost: content logging, backups, pinned image | You + agent | P0.08 | todo |
| P9.07 | Observability: telemetry, scrubbing, alerts, dashboards | You + agent | Phases 3–6 | todo |
| P9.08 | Canary, synthetics and status page | You + agent | P9.07, P3.11 | todo |
| P9.09 | Runbooks | Agent | P9.07 | todo |
| P9.10 | Load test and the production database tier | You + agent | Phases 3–6 | todo |
| P9.11 | Security review and open policies | You + agent | P9.01–P9.06 | todo |
| P9.12 | Terms, privacy and storage estimate | You + agent | P5.06 | todo |
| P9.13 | Developer docs live (Track D gate) | You + agent | D.09 | todo |
| P9.14 | Production build-out and staff verification (M3) | You + agent | All above | todo |
P9.01 — Droplet lockdown
Section titled “P9.01 — Droplet lockdown”Who: You + agent · Needs: P0.10, P0.09
In order:
- Set up WireGuard: the droplet peer and your laptop peer, with keys in Bitwarden.
- Confirm SSH, kubectl and the Bifrost dashboard work over it.
- Test the DigitalOcean web console as the recovery path.
- Create the DigitalOcean Cloud Firewall: inbound only the WireGuard UDP port, outbound open. The Niobi rule comes in P9.02.
- Close public SSH, 80 and 443. Stop relying on
ufw. - Remove the
prod-bifrost-core.router.africaingress and DNS record. - Add the scheduled external port scan.
- Spec: decisions log, Droplet network lockdown;
../infra/installation-as-built.md(planned changes at the top);../architecture/observability.md5.5 - Done when: an external scan shows nothing open except the WireGuard port (which doesn’t answer), and staging and production Workers still reach Bifrost and
niobi-proxy. Updateinstallation-as-built.mdto the new as-built state.
P9.02 — Niobi webhook path in production
Section titled “P9.02 — Niobi webhook path in production”Who: You + agent · Needs: P9.01, P6.03
-
You:
- pick the webhook hostname
- add a DNS-only (not proxied) record
- give Niobi the callback URL
- add the firewall rule allowing 443 only from Niobi’s source IPs
-
Agent:
- cert-manager with a DNS-01 issuer, using the DNS-edit token
- a Traefik route that forwards only the webhook path to
niobi-proxy
-
Spec: decisions log, Niobi webhooks;
../architecture/network-security-map.md(A6) -
Done when:
- a request from a non-Niobi IP gets no answer
- a Niobi test callback reaches the queue
- the certificate renews by DNS-01
P9.03 — Kubernetes network policies
Section titled “P9.03 — Kubernetes network policies”Who: Agent · Needs: P0.10, P6.01
Default-deny in every namespace. Allow only:
-
cloudflared→ the gateway proxy andniobi-proxy -
Traefik →
niobi-proxy(webhook) -
the gateway proxy → Bifrost
-
Bifrost → its Postgres
-
cloudflared→ the staging Postgres (Hyperdrive’s path), and the runner → the staging Postgres (migrations) -
each pod’s required egress
-
Spec: decisions log, Kubernetes network policies
-
Done when: from a
niobi-proxypod, Bifrost’s Postgres is unreachable, and every allowed path still works (staging and production smoke pass).
P9.04 — Cloudflare Access in production
Section titled “P9.04 — Cloudflare Access in production”Who: You · Needs: P8.01
Put admin.router.africa behind Access: a named allow-list (you) with the email one-time PIN. Create an Access service token for the Datadog admin probe.
- Spec: decisions log, Admin origin access
- Done when: an unlisted email can’t load the admin app, and the probe passes with its token.
P9.05 — Edge rules: HTTP refusal and production rate limits
Section titled “P9.05 — Edge rules: HTTP refusal and production rate limits”Who: You + agent · Needs: P8.07
-
HTTP refusal: choose and apply the Cloudflare rule that refuses plain HTTP on
api.router.africainstead of redirecting it, and record the choice. -
Rate limits: set the production rate-limit values in the admin app.
-
Spec:
../architecture/api-key-security.mdsections 6 and 10; decisions log, Edge rate limiting -
Done when:
curl http://api.router.africa/v1/modelsis refused without a redirect, and the production rules show in admin screen 55.
P9.06 — Bifrost: content logging, backups, pinned image
Section titled “P9.06 — Bifrost: content logging, backups, pinned image”Who: You + agent · Needs: P0.08
-
Content logging: confirm
disable_content_loggingis set, and verify no prompt content lands in Bifrost’s Postgres. -
Backups: a
pg_dumpCronJob to off-cluster storage (e.g. DigitalOcean Spaces), plus one restore test. -
Image: confirm the pinned image is recorded and a copy is kept.
-
Spec: backlog #17;
../infra/installation-as-built.mdremaining work, item 3; decisions log, Gateway abstraction (pinning) -
Done when: a test prompt leaves no content in Bifrost’s database, a backup from the last 24 hours exists, and a restore into a scratch namespace works.
P9.07 — Observability: telemetry, scrubbing, alerts, dashboards
Section titled “P9.07 — Observability: telemetry, scrubbing, alerts, dashboards”Who: You + agent · Needs: phases 3–6
-
Telemetry: Datadog through the Workers OpenTelemetry export, plus the droplet agent.
-
Spans: custom spans for auth, hold, the gateway call, settle and the queue publish.
-
Scrubbing:
Authorization, cookies, keys and prompt content are scrubbed, and that is verified. -
Alerts: every signal in
observability.mdsection 5, with its tier. -
Dashboards: the dashboards in section 7.
-
Deploy markers: sent from the pipeline.
-
On-call routing: decide it (it isn’t decided yet) and record it.
-
Spec:
../architecture/observability.md; backlog #17 Datadog -
Done when: a search in Datadog for a test prompt and a test key finds nothing, and each P1 alert has fired once in a staging drill.
P9.08 — Canary, synthetics and status page
Section titled “P9.08 — Canary, synthetics and status page”Who: You + agent · Needs: P9.07, P3.11
-
Canary: the P3.11 request every 5 minutes in production, on an internal workspace.
-
Synthetics:
api.andapp.probes from several regions, and the admin probe through Access. -
Status page: decide the tool, then publish the status page, fed by the probes and the canary. Left open by the operator on 2026-11-07, to decide here. Standing proposal: a hosted status-page service on a
status.subdomain, not something we run ourselves. Reasons: it must stay up when our stack is down (so not on the 4 GB droplet, and not on the same Workers asapi.), Datadog already runs the probes and can notify it by webhook, and it adds no server to operate. Shortlist to compare on price, custom domain on the free or low tier, webhook or API updates, and subscriber emails: Instatus, Better Stack, Atlassian Statuspage. A hand-built static page on Cloudflare Pages is the cheapest fallback but means updating it ourselves. The page is public, so it lists onlyapi.andapp., neveradmin.. Addingstatus.is an extra hostname outside the four-origin split; record it in the decisions log when chosen. -
Spec:
../architecture/observability.mdsection 6;../product/api-surface.mdsection 5 -
Done when: the status page is live and reflects a forced canary failure in staging.
P9.09 — Runbooks
Section titled “P9.09 — Runbooks”Who: Agent · Needs: P9.07
Write the nine runbooks listed in observability.md section 9, plus the leaked-key runbook in api-key-security.md 7.3. Each one covers: how to detect it, first steps, the commands, and how to communicate with customers.
- Spec:
../architecture/observability.mdsection 9;../architecture/api-key-security.md7.3 - Done when: each alert in section 5 links to its runbook.
P9.10 — Load test and the production database tier
Section titled “P9.10 — Load test and the production database tier”Who: You + agent · Needs: phases 3–6
k6 against the production database (the cluster Postgres) before any customer is on it, at the stated peak (about 1,000 writes per second), with hold/settle on one hot workspace and spread across many. Watch the Hyperdrive pool, Postgres latency, the droplet’s CPU and memory (Bifrost shares it) and the added Worker latency.
-
Decide: whether the cluster Postgres holds the launch load or production moves to PlanetScale before launch (decisions log, Ledger database), and the Hyperdrive pool size.
-
Tune: the alert thresholds that were waiting on load-test data.
-
Spec: decisions log, Ledger database;
../architecture/observability.md(thresholds to tune);../architecture/future-optimizations.mdLedger tier upgrade -
Done when: the database decision is recorded, and the p95 added latency is within the observability target at peak.
P9.11 — Security review and open policies
Section titled “P9.11 — Security review and open policies”Who: You + agent · Needs: P9.01–P9.06
-
Re-run the checks: both conformance suites, the header test against production hostnames, and the secret scan over the full history.
-
Walk the network map: every amber item is either closed or accepted in writing.
-
Decide the policy for credits spent through a leaked key, and the remaining items in
api-key-security.mdsection 10. -
Workers VPC: check its pricing and status.
-
Secrets: confirm the inventory is complete.
-
Spec:
../architecture/network-security-map.md;../architecture/api-key-security.mdsection 10;../infra/secrets-inventory.md -
Done when: the network map is updated, the policies are recorded in the decisions log, and every secret in use has an inventory row.
P9.12 — Terms, privacy and storage estimate
Section titled “P9.12 — Terms, privacy and storage estimate”Who: You + agent · Needs: P5.06
-
Website repo: terms of service and privacy policy that say plainly that prompts and responses are not saved unless the customer opts in, the retention of what is saved, and who can read it.
-
Storage: estimate R2 growth against expected traffic, given the indefinite retention.
-
Spec: backlog #17 Website, Storage estimate; decisions log, Audit trail & prompt/response retention
-
Done when: both documents are published on
router.africa, sign-up links to them, and the storage estimate is recorded in backlog #17.
P9.13 — Developer docs live (Track D gate)
Section titled “P9.13 — Developer docs live (Track D gate)”Who: You + agent · Needs: D.09
The developer docs are built in their own track, track-d-developer-docs.md: a custom docs app, the curated OpenAPI spec, the Request Builder and the AI-readable outputs. This task is done when that track’s launch gate, D.09, is done. D.10 (Ask AI) is not required.
- Spec:
track-d-developer-docs.md - Done when: D.09 is done.
P9.14 — Production build-out and staff verification (M3)
Section titled “P9.14 — Production build-out and staff verification (M3)”Who: You + agent · Needs: all above
- You: add production provider keys to Bifrost, over WireGuard.
- You: set every production secret per the inventory.
- You + agent: restore drill. Restore the production database from R2 to a point in time into a scratch namespace, and check the ledger matches. If P9.10 chose PlanetScale, do the move here instead, then run the same check against it.
- Cut
release/0.9and run Deploy to production forv0.9.0. - In a staff-only workspace:
- top up a small real amount through Niobi
- make real requests
- check the charge, the ledger, the reconciliations (both clean), the margin report and the alerts
- Spec:
../architecture/cicd.mdsections 1, 6 and 8 - Done when: every check above passes in production. That is M3.