Skip to content

Router.Africa — CI/CD and Deployment

Backlog item #13. How a change gets from a pull request to a running Worker, Pages site, K3s service or gateway config change. It restates the CI/CD pipeline, Release and branch flow, Migrations and the staging database, Container registry and Configuration and secrets management entries in decisions-log.md and adds the operating detail: gates, ordering rules, migrations, secrets and rollback. Where the two disagree, the decisions log wins.

Revised 2026-10-26. One pipeline on GitHub Actions replaces the earlier split between Cloudflare Workers Builds and GitHub Actions, and a release-branch flow replaces the proposed staging → main flow (decisions log, CI/CD pipeline, Release and branch flow).


Everything is deployed by GitHub Actions. Cloudflare only receives wrangler commands; Workers Builds and the Pages Git integration are not connected.

Target Runs on Environments Deployed by
apps/api Cloudflare Workers -staging, -production (one Wrangler [env.*] block each: own bindings and secrets) wrangler deploy --env <env>
apps/dashboard-api (also the queue() consumer and the scheduled-jobs dispatcher) Cloudflare Workers same same
apps/dashboard Cloudflare Pages production, staging, and a preview per PR wrangler pages deploy (--branch for previews). Deploys together with apps/dashboard-api
apps/admin Cloudflare Pages, its own project, behind Cloudflare Access staging, production wrangler pages deploy
apps/niobi-proxy K3s on the droplet Helm namespaces staging, production Image built and pushed to GHCR, then helm upgrade from the self-hosted runner in the cluster
Gateway config (infra/gateways/<instance>/) K3s on the droplet same namespaces Same runner and Helm path
  1. Gates (section 2).
  2. Database migration with the migrator role (section 4).
  3. apps/dashboard-api.
  4. apps/dashboard.
  5. apps/admin.
  6. apps/api.
  7. Droplet services: build and push the image (GitHub-hosted runner), then Helm deploy (self-hosted runner).
  8. Smoke check (section 6).
Event Runs Against
Pull request into main Gates (without the real-Postgres job), plus a dashboard preview Nothing is deployed except the preview
Merge to main The full pipeline, automatically Staging
Run workflow on Deploy to production (Actions tab, or gh workflow run deploy-production.yml -f tag=v1.4.0), given a release tag The full pipeline for that tag. Refused unless the tag is vX.Y.Z, exists, and points at a commit on main or release/*, so a tag on an unreviewed branch can’t ship Production
Run workflow on Gates (any branch) The gates, with the real-Postgres job on by default Nothing
Weekly Full-history secret scan; Dependabot (below) Nothing

Least privilege in the pipeline (code #38, review fixes). Every job runs npm ci, so a dependency’s install script sees whatever that job holds. So packages: write is granted only to the image-push job, and the Cloudflare deploy token only to the four jobs that deploy to Cloudflare. A PR’s own code runs in the preview job, so the preview uses its own Pages-only token (CLOUDFLARE_PAGES_PREVIEW_TOKEN, ../infra/secrets-inventory.md) and never holds the deploy token; until that token exists the preview is skipped with a notice. Workflow inputs reach shell scripts through env, never pasted into the script text.

Dependency updates (Dependabot). Weekly, for npm and GitHub Actions. The auth stack (better-auth, @better-auth/*, @simplewebauthn/*) gets its own PR, so its bump runs the gates (and the auth conformance suite) alone; other minor and patch bumps are grouped. No automatic major versions (code #91): each is upgraded deliberately on its own branch. Dependabot alerts still flag a vulnerable version.

There is no deployment-approval step: the repo is private on GitHub Free, where environment approvals aren’t available, and only the operator deploys at launch. The manual run is the deliberate step.

The droplet accepts no inbound connections after the lockdown, so nothing pushes into it. A self-hosted GitHub Actions runner (infra/ci-runner, P0.09) runs inside the cluster and connects out to GitHub. Its minutes don’t count against the free allowance, so every job of a deploy runs on it, the gates included (revised 2026-11-05; decisions log, Self-hosted runner and droplet deploys).

  • Which jobs: a job-started hook on the runner accepts only deploy-staging.yml and deploy-production.yml as committed on main, and refuses pull requests, other branches, tags and other workflows before any step runs. PR checks and manual Gates runs stay on GitHub-hosted runners.
  • One pod, one job at a time: the runner plus a throwaway Postgres 17 for the real-Postgres gate (no Docker on the runner, so no service containers). Low requests, capped memory (1280 Mi and 320 Mi) and the lowest priority on the node, so under memory pressure it’s evicted before production. On the runner, turbo runs one package at a time.
  • Kubernetes access: get and patch on named objects only: route-split’s config map, services and deployment in default, the cloudflared deployment, and the db-backup CronJob in postgres-production. No Secrets anywhere. Helm keeps its release records as Secrets, so the pipeline renders these charts with helm template and applies them with kubectl (infra/deploy-droplet.sh) instead of helm upgrade.
  • Shared releases change with production only. One cluster serves both environments, and route-split fronts production Bifrost, so route-split, cloudflared and db-backup are applied by a production deploy; a staging deploy prints what would change. Production Bifrost stays a manual helm upgrade (its bundled Postgres restarts on any change); niobi-proxy joins in phase 6.
  • Database access: the migrator URLs for both environments, for the migrate job (../infra/secrets-inventory.md).
  • Not on it: the niobi-proxy image build needs Docker, so it runs on a GitHub-hosted runner, only once phase 6 turns it on (NIOBI_PROXY_IMAGE).

What must pass before code is deployed.

Gate Runs Blocks
Lint, type check Every PR and every pipeline run Merge, deploy
Unit tests (Vitest) Same Merge, deploy
Worker integration tests (@cloudflare/vitest-pool-workers) Same Merge, deploy
DB integration tests on PGlite (Postgres 17 in WebAssembly) with the real migrations Same Merge, deploy
Real-Postgres job: the whole dashboard-api suite on a Postgres 17 service container (same major version as production), and the hold/settle concurrency test (N concurrent requests, one workspace, assert no over-approval; the decisions log calls it the highest-priority test) Every deploy (staging on each merge to main, and production), and a manual Gates run on any branch. Not on PRs (operator answer 14, 2026-11-01; code #54) Deploy
Auth conformance suite (auth-security.md section 6, plus the gateway management test) Every run, and required on every dependency bump Merge, deploy
Playwright end-to-end (top-up and balance first) After every staging deploy, against staging; nightly Cutting a release
k6 load test Not per PR. Before launch, and before any change to the hold/settle path or the ledger tier. Watched number: Hyperdrive pool size Launch sign-off, not a pipeline gate
Redaction and secret scan (no keys, cookies or content in logs; no secrets in the repo). A PR scans its own commits; pushes and a weekly run scan the full history. The sk-ra- rule needs entropy above 3.5, so made-up test keys pass (decisions log, Code review fixes, P0–P4) Every PR, every push, weekly Merge

Minutes: GitHub Free gives 2,000 hosted-runner minutes a month for a private repo; they ran out on 2026-11-04. Deploys now use none (section 1, Reaching the cluster). PR checks still do: when the month’s minutes run out they don’t start until the next month, while merges and deploys carry on, and every deploy runs the full gates itself. End-to-end and load suites, if they outgrow that, move to a self-hosted runner on a small separate machine, not the production droplet. Jenkins is not used (decisions log, CI/CD pipeline, and Self-hosted runner and droplet deploys).

Several pieces are deployed by different systems at different moments, so a change must be safe with the old and the new version running side by side. These are the rules that follow from the architecture.

  1. Schema first, code second, and both compatible. Migrations live in packages/db and the pipeline applies them before deploying any code. A migration must work with the code currently deployed (add columns nullable or defaulted, add before use, remove only after nothing reads them). Both Workers read the same tables from the same commit, and they don’t deploy at the same instant.
  2. Queue events are versioned. apps/api publishes usage events and apps/dashboard-api consumes them, and they deploy independently. Events carry a version; the consumer accepts the current and the previous version; deploy the consumer first when the shape changes. The consumer is idempotent (decisions log, Log-write and audit rules), so redelivery during a deploy is safe.
  3. Dashboard and dashboard-api change together but stay compatible. They are one deploy unit (decided) and the pipeline deploys dashboard-api first, then the dashboard, but for a short window one can still be ahead of the other. API changes are additive; removals ship in a later deploy.
  4. Scheduled jobs survive a deploy. A deploy of apps/dashboard-api can land while a job runs. Jobs hold a lease and are idempotent (watermarks advance only after a window is fully read), and a dead run’s lease is reclaimed. Nothing in the pipeline needs to pause the dispatcher.
  5. Bifrost changes go through key sync. A Bifrost upgrade or config change can change the shape of virtual key fields (allowed_models, rate limits, budgets; backlog #18 lists these as still to verify). After a Bifrost deploy, run the key reconciliation on staging and confirm the discrepancy view is empty before releasing to production.
  6. Key-auth caching needs a revocation path. Revocation is immediate because apps/api reads the key from Postgres on every request. If a cache is ever added in front of that lookup, it needs a bound on staleness and a way to invalidate on revoke (see api-key-security.md).
  • Migrations are versioned Drizzle files in packages/db, reviewed in the PR like code. Role grants (decisions log, Database roles) are migrations too.
  • CI applies them to a fresh Postgres container to run the DB integration tests, which proves they apply cleanly from scratch.
  • The pipeline applies them with the migrator role as its first deploy step: to the staging database on every merge to main, to production when a release is deployed. Both databases are self-hosted Postgres in the K3s cluster (production moves to PlanetScale later; decisions log, Ledger database), so the migrate step reaches both through the self-hosted runner.
  • Expand, then contract. Two releases for anything destructive (rename, drop, type change), never one.
  • Large-table changes (logs.request_logs) follow the future-optimizations hooks: keys include created_at, no foreign keys to the ledger, so a partitioning migration stays possible.
  • Forward only. Databases are not rolled back; a bad migration is fixed with a new one, which is why ordering rule 1 matters.
  • Hotfixes avoid migrations. An unavoidable one lands on main first and is cherry-picked unchanged into the release branch, so both lines apply the same file.
Item Rule
Separation Staging and production have separate Workers, bindings, Hyperdrive configs, queues, R2 buckets, cookies, databases and secrets. A staging credential never works in production and the reverse
Databases Self-hosted Postgres in the K3s cluster (infra/postgres/, CloudNativePG), one instance per environment in its own namespace, separate from Bifrost’s Postgres. Staging is disposable and reseeded; production backs up continuously to R2. Production moves to PlanetScale later
Staging Niobi apps/niobi-proxy runs in mock mode (environment flag), so staging needs no live payment credentials and can’t move real money
Staging gateway Its own Helm namespace on the same droplet (infra/gateways/), its own Workers VPC Services and bindings; provider keys for staging are separate from production or absent
Non-secret config In the repo: [env.staging] / [env.production] in wrangler.toml, values-staging.yaml / values-production.yaml for Helm. Deployed by the pipeline
Master copy of secrets A Bitwarden vault (free plan), one entry per secret per environment. Also holds account MFA recovery codes, WireGuard keys and the off-server copy of Bifrost’s encryption key
Where secrets are deployed Worker secrets (wrangler secret put), Kubernetes Secrets (kubectl, over WireGuard), and GitHub Actions repository secrets for what CI needs only (today only the Cloudflare deploy token, plus the account id as a variable; database credentials stay in the cluster), prefixed STAGING_ / PROD_ only if a value differs per environment. Never the repo
Inventory ../infra/secrets-inventory.md: every secret by name, where it’s deployed, who holds it, how to rotate it. Never values
Auth secrets BETTER_AUTH_SECRET_APP and BETTER_AUTH_SECRET_ADMIN differ per environment and per audience (auth-security.md 3.1)
Niobi credentials and signing Only in apps/niobi-proxy secrets (containment requirement). No other service’s config contains them
Rotation Every secret has a documented rotation step in the inventory. Breach response includes deleting every row in sessions or rotating the auth secrets (auth-security.md 3.2)
Upgrade path When a second person needs access: Bitwarden Secrets Manager (free tier), read directly by CI
  • Smoke check: health of api., app. and admin., and one end-to-end request through hold, Bifrost and settle against a dedicated test workspace on that environment. The same request, run on a schedule in production, is the synthetic canary in observability.md.
  • Watch window: for the first 15 minutes after a production deploy, the alerts in observability.md for 5xx rate, settle failures and queue lag are what tell you to roll back (deploy markers are sent to Datadog so a change lines up with a graph).
  • Staging-only verifications listed in backlog #18 (the x-request-id behaviour, GET /api/logs, the model listing endpoint) run before the usage reconciliation is relied on in production.
Target Rollback Note
Workers Run the production workflow for the previous release tag, or roll back to the previous version in Cloudflare Instant. Check the queue consumer and cron still match the events in flight
Pages Promote the previous deployment Instant
K3s services and Bifrost helm rollback to the previous release Then run the key reconciliation if the change touched virtual keys
Database None. Forward-only Fix forward, with code that works on the old and new schema
Secrets Restore the previous value and redeploy Auth secret rotation signs everyone out
  • main is the only long-lived branch. PRs merge into main; every merge deploys to staging.
  • Releases use semantic versions (major.minor.patch). Cut release/1.4 from main, tag v1.4.0, run Deploy to production for the tag.
  • Hotfix: merge the fix to main first, cherry-pick it into release/1.4, tag v1.4.1, run Deploy to production. Fixing main first means the next release can’t drop the fix.
  • Protection: not enforced by GitHub (GitHub Free, private repo; decisions log, Release and branch flow). By rule, main and release/* take no direct pushes and change only through PRs that ran the section 2 checks.
  • Dependencies pinned to exact versions for the auth stack (Better Auth and its plugins); an automated update PR runs the conformance suite before merge.
  • Turborepo caches lint, test and build per package, so a change to apps/dashboard doesn’t re-test apps/api unless a shared package changed.
  • GHCR’s free storage and transfer allowance for a private image, and the cleanup policy that keeps within it.
  • How Cloudflare’s rate-limiting rules are counted on the chosen plan (decisions log, Edge rate limiting).
  • Whether the hold/settle concurrency test is fast enough to run on every PR.
  • The self-hosted runner’s install method (a single runner pod, or actions-runner-controller) and its resource limits on the droplet.