Router.Africa — CI/CD and Deployment
Backlog item #13. How a change gets from a pull request to a running Worker, Pages site, K3s service or gateway config change. It restates the CI/CD pipeline, Release and branch flow, Migrations and the staging database, Container registry and Configuration and secrets management entries in decisions-log.md and adds the operating detail: gates, ordering rules, migrations, secrets and rollback. Where the two disagree, the decisions log wins.
Revised 2026-10-26. One pipeline on GitHub Actions replaces the earlier split between Cloudflare Workers Builds and GitHub Actions, and a release-branch flow replaces the proposed
staging→mainflow (decisions log, CI/CD pipeline, Release and branch flow).
1. What deploys where
Section titled “1. What deploys where”Everything is deployed by GitHub Actions. Cloudflare only receives wrangler commands; Workers Builds and the Pages Git integration are not connected.
| Target | Runs on | Environments | Deployed by |
|---|---|---|---|
apps/api |
Cloudflare Workers | -staging, -production (one Wrangler [env.*] block each: own bindings and secrets) |
wrangler deploy --env <env> |
apps/dashboard-api (also the queue() consumer and the scheduled-jobs dispatcher) |
Cloudflare Workers | same | same |
apps/dashboard |
Cloudflare Pages | production, staging, and a preview per PR | wrangler pages deploy (--branch for previews). Deploys together with apps/dashboard-api |
apps/admin |
Cloudflare Pages, its own project, behind Cloudflare Access | staging, production | wrangler pages deploy |
apps/niobi-proxy |
K3s on the droplet | Helm namespaces staging, production |
Image built and pushed to GHCR, then helm upgrade from the self-hosted runner in the cluster |
Gateway config (infra/gateways/<instance>/) |
K3s on the droplet | same namespaces | Same runner and Helm path |
The pipeline, in order
Section titled “The pipeline, in order”- Gates (section 2).
- Database migration with the
migratorrole (section 4). apps/dashboard-api.apps/dashboard.apps/admin.apps/api.- Droplet services: build and push the image (GitHub-hosted runner), then Helm deploy (self-hosted runner).
- Smoke check (section 6).
Triggers
Section titled “Triggers”| Event | Runs | Against |
|---|---|---|
Pull request into main |
Gates (without the real-Postgres job), plus a dashboard preview | Nothing is deployed except the preview |
Merge to main |
The full pipeline, automatically | Staging |
Run workflow on Deploy to production (Actions tab, or gh workflow run deploy-production.yml -f tag=v1.4.0), given a release tag |
The full pipeline for that tag. Refused unless the tag is vX.Y.Z, exists, and points at a commit on main or release/*, so a tag on an unreviewed branch can’t ship |
Production |
| Run workflow on Gates (any branch) | The gates, with the real-Postgres job on by default | Nothing |
| Weekly | Full-history secret scan; Dependabot (below) | Nothing |
Least privilege in the pipeline (code #38, review fixes). Every job runs npm ci, so a dependency’s install script sees whatever that job holds. So packages: write is granted only to the image-push job, and the Cloudflare deploy token only to the four jobs that deploy to Cloudflare. A PR’s own code runs in the preview job, so the preview uses its own Pages-only token (CLOUDFLARE_PAGES_PREVIEW_TOKEN, ../infra/secrets-inventory.md) and never holds the deploy token; until that token exists the preview is skipped with a notice. Workflow inputs reach shell scripts through env, never pasted into the script text.
Dependency updates (Dependabot). Weekly, for npm and GitHub Actions. The auth stack (better-auth, @better-auth/*, @simplewebauthn/*) gets its own PR, so its bump runs the gates (and the auth conformance suite) alone; other minor and patch bumps are grouped. No automatic major versions (code #91): each is upgraded deliberately on its own branch. Dependabot alerts still flag a vulnerable version.
There is no deployment-approval step: the repo is private on GitHub Free, where environment approvals aren’t available, and only the operator deploys at launch. The manual run is the deliberate step.
Reaching the cluster
Section titled “Reaching the cluster”The droplet accepts no inbound connections after the lockdown, so nothing pushes into it. A self-hosted GitHub Actions runner (infra/ci-runner, P0.09) runs inside the cluster and connects out to GitHub. Its minutes don’t count against the free allowance, so every job of a deploy runs on it, the gates included (revised 2026-11-05; decisions log, Self-hosted runner and droplet deploys).
- Which jobs: a job-started hook on the runner accepts only
deploy-staging.ymlanddeploy-production.ymlas committed onmain, and refuses pull requests, other branches, tags and other workflows before any step runs. PR checks and manual Gates runs stay on GitHub-hosted runners. - One pod, one job at a time: the runner plus a throwaway Postgres 17 for the real-Postgres gate (no Docker on the runner, so no service containers). Low requests, capped memory (1280 Mi and 320 Mi) and the lowest priority on the node, so under memory pressure it’s evicted before production. On the runner, turbo runs one package at a time.
- Kubernetes access: get and patch on named objects only:
route-split’s config map, services and deployment indefault, thecloudflareddeployment, and thedb-backupCronJob inpostgres-production. No Secrets anywhere. Helm keeps its release records as Secrets, so the pipeline renders these charts withhelm templateand applies them withkubectl(infra/deploy-droplet.sh) instead ofhelm upgrade. - Shared releases change with production only. One cluster serves both environments, and
route-splitfronts production Bifrost, soroute-split,cloudflaredanddb-backupare applied by a production deploy; a staging deploy prints what would change. Production Bifrost stays a manualhelm upgrade(its bundled Postgres restarts on any change);niobi-proxyjoins in phase 6. - Database access: the migrator URLs for both environments, for the migrate job (
../infra/secrets-inventory.md). - Not on it: the
niobi-proxyimage build needs Docker, so it runs on a GitHub-hosted runner, only once phase 6 turns it on (NIOBI_PROXY_IMAGE).
2. Gates
Section titled “2. Gates”What must pass before code is deployed.
| Gate | Runs | Blocks |
|---|---|---|
| Lint, type check | Every PR and every pipeline run | Merge, deploy |
| Unit tests (Vitest) | Same | Merge, deploy |
Worker integration tests (@cloudflare/vitest-pool-workers) |
Same | Merge, deploy |
| DB integration tests on PGlite (Postgres 17 in WebAssembly) with the real migrations | Same | Merge, deploy |
Real-Postgres job: the whole dashboard-api suite on a Postgres 17 service container (same major version as production), and the hold/settle concurrency test (N concurrent requests, one workspace, assert no over-approval; the decisions log calls it the highest-priority test) |
Every deploy (staging on each merge to main, and production), and a manual Gates run on any branch. Not on PRs (operator answer 14, 2026-11-01; code #54) |
Deploy |
Auth conformance suite (auth-security.md section 6, plus the gateway management test) |
Every run, and required on every dependency bump | Merge, deploy |
| Playwright end-to-end (top-up and balance first) | After every staging deploy, against staging; nightly | Cutting a release |
| k6 load test | Not per PR. Before launch, and before any change to the hold/settle path or the ledger tier. Watched number: Hyperdrive pool size | Launch sign-off, not a pipeline gate |
Redaction and secret scan (no keys, cookies or content in logs; no secrets in the repo). A PR scans its own commits; pushes and a weekly run scan the full history. The sk-ra- rule needs entropy above 3.5, so made-up test keys pass (decisions log, Code review fixes, P0–P4) |
Every PR, every push, weekly | Merge |
Minutes: GitHub Free gives 2,000 hosted-runner minutes a month for a private repo; they ran out on 2026-11-04. Deploys now use none (section 1, Reaching the cluster). PR checks still do: when the month’s minutes run out they don’t start until the next month, while merges and deploys carry on, and every deploy runs the full gates itself. End-to-end and load suites, if they outgrow that, move to a self-hosted runner on a small separate machine, not the production droplet. Jenkins is not used (decisions log, CI/CD pipeline, and Self-hosted runner and droplet deploys).
3. Ordering rules
Section titled “3. Ordering rules”Several pieces are deployed by different systems at different moments, so a change must be safe with the old and the new version running side by side. These are the rules that follow from the architecture.
- Schema first, code second, and both compatible. Migrations live in
packages/dband the pipeline applies them before deploying any code. A migration must work with the code currently deployed (add columns nullable or defaulted, add before use, remove only after nothing reads them). Both Workers read the same tables from the same commit, and they don’t deploy at the same instant. - Queue events are versioned.
apps/apipublishes usage events andapps/dashboard-apiconsumes them, and they deploy independently. Events carry a version; the consumer accepts the current and the previous version; deploy the consumer first when the shape changes. The consumer is idempotent (decisions log, Log-write and audit rules), so redelivery during a deploy is safe. - Dashboard and dashboard-api change together but stay compatible. They are one deploy unit (decided) and the pipeline deploys
dashboard-apifirst, then the dashboard, but for a short window one can still be ahead of the other. API changes are additive; removals ship in a later deploy. - Scheduled jobs survive a deploy. A deploy of
apps/dashboard-apican land while a job runs. Jobs hold a lease and are idempotent (watermarks advance only after a window is fully read), and a dead run’s lease is reclaimed. Nothing in the pipeline needs to pause the dispatcher. - Bifrost changes go through key sync. A Bifrost upgrade or config change can change the shape of virtual key fields (
allowed_models, rate limits, budgets; backlog #18 lists these as still to verify). After a Bifrost deploy, run the key reconciliation on staging and confirm the discrepancy view is empty before releasing to production. - Key-auth caching needs a revocation path. Revocation is immediate because
apps/apireads the key from Postgres on every request. If a cache is ever added in front of that lookup, it needs a bound on staleness and a way to invalidate on revoke (seeapi-key-security.md).
4. Database migrations
Section titled “4. Database migrations”- Migrations are versioned Drizzle files in
packages/db, reviewed in the PR like code. Role grants (decisions log, Database roles) are migrations too. - CI applies them to a fresh Postgres container to run the DB integration tests, which proves they apply cleanly from scratch.
- The pipeline applies them with the
migratorrole as its first deploy step: to the staging database on every merge tomain, to production when a release is deployed. Both databases are self-hosted Postgres in the K3s cluster (production moves to PlanetScale later; decisions log, Ledger database), so the migrate step reaches both through the self-hosted runner. - Expand, then contract. Two releases for anything destructive (rename, drop, type change), never one.
- Large-table changes (
logs.request_logs) follow the future-optimizations hooks: keys includecreated_at, no foreign keys to the ledger, so a partitioning migration stays possible. - Forward only. Databases are not rolled back; a bad migration is fixed with a new one, which is why ordering rule 1 matters.
- Hotfixes avoid migrations. An unavoidable one lands on
mainfirst and is cherry-picked unchanged into the release branch, so both lines apply the same file.
5. Environments and secrets
Section titled “5. Environments and secrets”| Item | Rule |
|---|---|
| Separation | Staging and production have separate Workers, bindings, Hyperdrive configs, queues, R2 buckets, cookies, databases and secrets. A staging credential never works in production and the reverse |
| Databases | Self-hosted Postgres in the K3s cluster (infra/postgres/, CloudNativePG), one instance per environment in its own namespace, separate from Bifrost’s Postgres. Staging is disposable and reseeded; production backs up continuously to R2. Production moves to PlanetScale later |
| Staging Niobi | apps/niobi-proxy runs in mock mode (environment flag), so staging needs no live payment credentials and can’t move real money |
| Staging gateway | Its own Helm namespace on the same droplet (infra/gateways/), its own Workers VPC Services and bindings; provider keys for staging are separate from production or absent |
| Non-secret config | In the repo: [env.staging] / [env.production] in wrangler.toml, values-staging.yaml / values-production.yaml for Helm. Deployed by the pipeline |
| Master copy of secrets | A Bitwarden vault (free plan), one entry per secret per environment. Also holds account MFA recovery codes, WireGuard keys and the off-server copy of Bifrost’s encryption key |
| Where secrets are deployed | Worker secrets (wrangler secret put), Kubernetes Secrets (kubectl, over WireGuard), and GitHub Actions repository secrets for what CI needs only (today only the Cloudflare deploy token, plus the account id as a variable; database credentials stay in the cluster), prefixed STAGING_ / PROD_ only if a value differs per environment. Never the repo |
| Inventory | ../infra/secrets-inventory.md: every secret by name, where it’s deployed, who holds it, how to rotate it. Never values |
| Auth secrets | BETTER_AUTH_SECRET_APP and BETTER_AUTH_SECRET_ADMIN differ per environment and per audience (auth-security.md 3.1) |
| Niobi credentials and signing | Only in apps/niobi-proxy secrets (containment requirement). No other service’s config contains them |
| Rotation | Every secret has a documented rotation step in the inventory. Breach response includes deleting every row in sessions or rotating the auth secrets (auth-security.md 3.2) |
| Upgrade path | When a second person needs access: Bitwarden Secrets Manager (free tier), read directly by CI |
6. After a deploy
Section titled “6. After a deploy”- Smoke check: health of
api.,app.andadmin., and one end-to-end request through hold, Bifrost and settle against a dedicated test workspace on that environment. The same request, run on a schedule in production, is the synthetic canary inobservability.md. - Watch window: for the first 15 minutes after a production deploy, the alerts in
observability.mdfor 5xx rate, settle failures and queue lag are what tell you to roll back (deploy markers are sent to Datadog so a change lines up with a graph). - Staging-only verifications listed in backlog #18 (the
x-request-idbehaviour,GET /api/logs, the model listing endpoint) run before the usage reconciliation is relied on in production.
7. Rollback
Section titled “7. Rollback”| Target | Rollback | Note |
|---|---|---|
| Workers | Run the production workflow for the previous release tag, or roll back to the previous version in Cloudflare | Instant. Check the queue consumer and cron still match the events in flight |
| Pages | Promote the previous deployment | Instant |
| K3s services and Bifrost | helm rollback to the previous release |
Then run the key reconciliation if the change touched virtual keys |
| Database | None. Forward-only | Fix forward, with code that works on the old and new schema |
| Secrets | Restore the previous value and redeploy | Auth secret rotation signs everyone out |
8. Branches and releases
Section titled “8. Branches and releases”mainis the only long-lived branch. PRs merge intomain; every merge deploys to staging.- Releases use semantic versions (
major.minor.patch). Cutrelease/1.4frommain, tagv1.4.0, run Deploy to production for the tag. - Hotfix: merge the fix to
mainfirst, cherry-pick it intorelease/1.4, tagv1.4.1, run Deploy to production. Fixingmainfirst means the next release can’t drop the fix. - Protection: not enforced by GitHub (GitHub Free, private repo; decisions log, Release and branch flow). By rule,
mainandrelease/*take no direct pushes and change only through PRs that ran the section 2 checks. - Dependencies pinned to exact versions for the auth stack (Better Auth and its plugins); an automated update PR runs the conformance suite before merge.
- Turborepo caches lint, test and build per package, so a change to
apps/dashboarddoesn’t re-testapps/apiunless a shared package changed.
9. To confirm at build
Section titled “9. To confirm at build”- GHCR’s free storage and transfer allowance for a private image, and the cleanup policy that keeps within it.
- How Cloudflare’s rate-limiting rules are counted on the chosen plan (decisions log, Edge rate limiting).
- Whether the hold/settle concurrency test is fast enough to run on every PR.
- The self-hosted runner’s install method (a single runner pod, or actions-runner-controller) and its resource limits on the droplet.