Router.Africa — Bifrost Installation (As-Built)
Status: Live in production
Server: [redacted] — DigitalOcean Droplet, [redacted-ip], Ubuntu 26.04 LTS
Gateway URL: https://prod-bifrost-core.router.africa
Bifrost version: v2.2.3 (chart bifrost-2.1.43)
Last updated: 2026-11-01 (P0.08 at 20:50 UTC)
Planned changes before public launch (decided 2026-10-25, not yet applied). This file stays the record of what is installed; these change it at the pre-launch lockdown (decisions log, Droplet network lockdown, Private service access, Niobi webhooks, Gateway abstraction):
- Firewall: move from
ufwto the DigitalOcean Cloud Firewall (K3s’s iptables rules can bypassufw). Inbound closed except the WireGuard UDP port and 443 from Niobi’s webhook IPs. SSH, 80 and public 443 closed. Test the DigitalOcean web console as the recovery path first.- Operator access: WireGuard for SSH, kubectl and the Bifrost dashboard.
- Bifrost: remove the
prod-bifrost-core.router.africaingress and DNS record. Workers reach Bifrost throughcloudflared(in-cluster, outbound only) and a Workers VPC Service. The Workers use that path from day one; the public hostname is for the operator during the build only.- Layout: Helm config moves to
infra/gateways/bifrost-1/in the code repo; pin the exact Bifrost image and keep a copy.- cert-manager: the Niobi webhook host uses the DNS-01 challenge (Cloudflare API token), since port 80 will be closed.
This documents what was actually installed and configured on this server, in order, so it can be reproduced or audited later. Rancher was deliberately not used — see rancher-setup-TODO.md for that deferred piece.
1. Server rebuild
Section titled “1. Server rebuild”The Droplet was rebuilt clean via the DigitalOcean control panel (Droplet page → Destroy tab → Rebuild) on image Ubuntu 26.04 LTS (“Resolute Raccoon”), keeping the same IP ([redacted-ip]).
2. Initial user setup
Section titled “2. Initial user setup”Logged in as root using the one-time password DigitalOcean emailed after the rebuild, then:
apt update && apt upgrade -yadduser bifrostusermod -aG sudo bifrostSSH key-based access was set up for bifrost via ssh-copy-id from the local machine (rather than copying root’s .ssh, since root had no key on it), and confirmed working with sudo whoami returning root.
3. Firewall
Section titled “3. Firewall”sudo ufw allow OpenSSHsudo ufw allow 80/tcpsudo ufw allow 443/tcpsudo ufw enablePort 6443 (Kubernetes API) is deliberately not opened to the internet. An IP-restricted rule was tried first, then removed once it became clear the operator isn’t on a static IP — kubectl is only used locally on the Droplet itself via SSH, so 6443 doesn’t need to be reachable externally at all. Current state, confirmed via sudo ufw status:
OpenSSH ALLOW Anywhere80/tcp ALLOW Anywhere443/tcp ALLOW Anywhere(+ IPv6 equivalents)4. K3s
Section titled “4. K3s”curl -sfL https://get.k3s.io | sh -Installed as v1.36.4+k3s1, single control-plane node, ships with Traefik (ingress) and ServiceLB built in.
kubectl configured for the bifrost user (not root, not sudo k3s kubectl):
mkdir -p ~/.kubesudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/configsudo chown $(id -u):$(id -g) ~/.kube/configexport KUBECONFIG=~/.kube/configecho 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc5. Helm
Section titled “5. Helm”curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bashInstalled: v3.22.0.
6. cert-manager
Section titled “6. cert-manager”helm repo add jetstack https://charts.jetstack.iohelm repo updatekubectl create namespace cert-managerhelm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=trueClusterIssuer for Let’s Encrypt:
cat > cluster-issuer.yaml << 'EOF'apiVersion: cert-manager.io/v1kind: ClusterIssuermetadata: name: letsencrypt-prodspec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: mesyedhassan@gmail.com privateKeySecretRef: name: letsencrypt-prod solvers: - http01: ingress: class: traefikEOF
kubectl apply -f cluster-issuer.yaml7. DNS
Section titled “7. DNS”A record added at the DNS provider managing router.africa (proxy/orange-cloud left off — required for the Let’s Encrypt HTTP-01 challenge to reach the Droplet directly):
Type: AName: prod-bifrost-coreValue: [redacted-ip]8. Secrets
Section titled “8. Secrets”ENCRYPTION_KEY=$(openssl rand -base64 32)kubectl create secret generic bifrost-encryption-key \ --from-literal=encryption-key="$ENCRYPTION_KEY"# (value saved outside the cluster separately)
kubectl create secret generic postgres-credentials \ --from-literal=password="$(openssl rand -base64 24)"Provider API keys (provider-keys secret) were not created yet — keys weren’t available at install time. See “Remaining work” below.
9. Bifrost Helm repo
Section titled “9. Bifrost Helm repo”helm repo add bifrost https://maximhq.github.io/bifrost/helm-chartshelm repo updateInstalled chart: bifrost-2.1.40, app version 1.5.12. (Upgraded to chart 2.1.43 / app v2.2.3 on 2026-09-27 — see Section 13.)
10. Helm values (final, working version)
Section titled “10. Helm values (final, working version)”This is the validated values.yaml actually deployed — reached after correcting two issues the first draft got wrong against the real chart schema (see “Issues hit during install” below):
image: tag: "v2.2.3" # originally v1.5.12 at install; upgraded 2026-09-27, see Section 13
replicaCount: 1
storage: mode: postgres persistence: enabled: false configStore: enabled: true logsStore: enabled: true
postgresql: enabled: true auth: username: bifrost database: bifrost existingSecret: "postgres-credentials" passwordKey: "password" primary: persistence: enabled: true size: 8Gi resources: requests: cpu: 250m memory: 256Mi limits: cpu: 500m memory: 1Gi
resources: requests: cpu: 250m memory: 256Mi limits: cpu: 500m memory: 512Mi
livenessProbe: httpGet: path: /health port: http initialDelaySeconds: 45 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 6
readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 45 periodSeconds: 5 timeoutSeconds: 5 failureThreshold: 6
ingress: enabled: true className: traefik annotations: cert-manager.io/cluster-issuer: letsencrypt-prod hosts: - host: prod-bifrost-core.router.africa paths: - path: / pathType: Prefix tls: - secretName: bifrost-tls hosts: - prod-bifrost-core.router.africa
bifrost: encryptionKeySecret: name: "bifrost-encryption-key" key: "encryption-key" client: allowedOrigins: - "https://router.africa" - "https://prod-bifrost-core.router.africa" enforceAuthOnInference: true
env: - name: GOGC value: "200" - name: GOMEMLIMIT value: "460MiB"Install command:
helm install bifrost bifrost/bifrost -f values.yaml11. Issues hit during install (and fixes)
Section titled “11. Issues hit during install (and fixes)”Schema mismatch on first attempt. helm install --dry-run caught this immediately:
Error: at '/bifrost/client': additional properties 'enforce_auth_on_inference', 'allowed_origins' not allowedFix: the chart uses camelCase (enforceAuthOnInference, allowedOrigins), not the snake_case used in raw config.json. Also, postgresql is a top-level values key in this chart, not nested under storage.postgresql as first assumed — the nested version wouldn’t have errored, it would have silently deployed nothing.
Missing required field. image.tag has no default and must be set explicitly — resolved by checking helm search repo bifrost/bifrost --versions for the current app version (1.5.12) and using tag v1.5.12.
CrashLoopBackOff after a clean dry-run and install. The pod alternated between Running and Error/CrashLoopBackOff. kubectl logs <pod> --previous showed the real cause: Bifrost logged "Time spent in Bifrost server bootstrap 33805 ms" and "successfully started", then 14 seconds later received SIGTERM and shut down gracefully — it wasn’t crashing internally, Kubernetes was killing it. The chart’s default livenessProbe (5s initial delay + 10s period × 3 failures = 35s total) was too tight against a ~34s real startup time (Postgres connection, config/logs store migrations, model catalog sync, MCP library sync).
First fix attempt (startupProbe) didn’t work — it wasn’t rejected by schema validation, but kubectl describe deployment confirmed the chart template doesn’t render a startupProbe field at all, so it was silently ignored. Actual fix: raised livenessProbe/readinessProbe initialDelaySeconds to 45 and failureThreshold to 6 (giving ~105s of margin) — these probes back the pod’s real health checks and cover it correctly, confirmed via kubectl describe deployment bifrost showing delay=45s ... #failure=6 on both.
12. Upgrade: v1.5.12 → v2.2.3 (2026-09-27)
Section titled “12. Upgrade: v1.5.12 → v2.2.3 (2026-09-27)”The instance had been running stable for ~12 days on v1.5.12 with no data beyond an admin account, so this upgrade was done without a Postgres backup first — an explicit call given the low stakes (fresh instance, nothing valuable to lose yet). This is not the standard going forward once real data exists; see “Remaining work” item 3 below.
Step 1 — refresh Helm repo as the right user. helm repo update failed under root (Error: no repositories found. You must add one before updating) because the bifrost Helm repo was only ever added under the bifrost user, not root. Fixed by switching user first:
su - bifrosthelm repo updatehelm search repo bifrost/bifrost --versionsThis listed chart versions 2.1.34 through 2.1.43, but all of them showed app version 1.5.12 in helm search output — the chart’s Chart.yaml appVersion field hadn’t been bumped in lockstep with actual Bifrost releases, so chart version was not a reliable proxy for the image tag.
Step 2 — verify the target image tag actually exists. Queried Docker Hub’s v2 API directly rather than trusting helm search:
curl -s "https://hub.docker.com/v2/repositories/maximhq/bifrost/tags?page_size=100" | grep -o '"name":"[^"]*"'A first pass returned tags in an unhelpful default order that made it look like v2.2.3 might not exist; re-running with explicit sorting/filtering on the tag name confirmed v2.2.3 is a real, published tag.
Step 3 — update values.yaml and dry-run.
sed -i 's/tag: "v1.5.12"/tag: "v2.2.3"/' values.yamlhelm upgrade bifrost bifrost/bifrost -f values.yaml --dry-runDry-run output confirmed: chart moves from 2.1.40 → 2.1.43, image tag from v1.5.12 → v2.2.3, no other schema changes needed — the values.yaml structure from the original install (Section 10) was still valid against the new chart version.
Step 4 — real upgrade.
helm upgrade bifrost bifrost/bifrost -f values.yamlkubectl get pods -wIssue hit: transient Postgres connection-refused on the new Bifrost pod. The new bifrost pod came up with 2 restarts. kubectl logs <pod> --previous showed:
failed to connect to `user=bifrost database=bifrost`: 10.43.139.154:5432 (bifrost-postgresql): dial error: dial tcp 10.43.139.154:5432: connect: connection refusedDiagnosis: the bundled Postgres sub-chart’s Deployment uses strategy: Recreate (not a rolling update), so helm upgrade tore down and recreated the Postgres pod at roughly the same time as the Bifrost pod. Bifrost’s new pod started faster than Postgres came back up, tried to connect, failed, and was restarted by Kubernetes — which then succeeded once Postgres was ready. Not a v2.2.3 regression, just an ordering race between two pods with no dependency/readiness gate between them at the Helm level.
Confirmed resolved (not an ongoing crash loop) by watching restart count hold steady rather than climb, then checking health:
kubectl get pods# bifrost-d49cdf95f-fcwxj 1/1 Running 2 (2m59s ago) 3m11s# bifrost-postgresql-5b7b668d6f-27976 1/1 Running 0 3m10s
curl https://prod-bifrost-core.router.africa/health# {"components":{"db_pings":"ok"},"status":"ok"}Takeaway for future upgrades: the Postgres-recreate race is expected behavior with this chart’s current settings, not a fault — a brief restart or two on the Bifrost pod right after an upgrade is normal and self-resolves once Postgres is back. If this needs to be eliminated (e.g. once there’s real traffic and even a few seconds of restart noise matters), options to revisit: pin the Postgres sub-chart to strategy: RollingUpdate if the chart exposes that, or add an init container / readiness gate on Bifrost that waits for Postgres before starting.
13. Verification
Section titled “13. Verification”kubectl get pods# bifrost-56df86bcb-ll2ln 1/1 Running 0 restarts# bifrost-postgresql-... 1/1 Running 0 restarts
kubectl get certificate# bifrost-tls READY: True
curl https://prod-bifrost-core.router.africa/health# {"components":{"db_pings":"ok"},"status":"ok"}Full chain confirmed working end to end: DNS → Let’s Encrypt/HTTPS → Traefik ingress → Bifrost → self-hosted Postgres.
14. Platform setup for the app (2026-11-01, P0.02 and P0.03)
Section titled “14. Platform setup for the app (2026-11-01, P0.02 and P0.03)”Droplet capacity: 2 vCPU, 4 GB RAM, 70 GB disk. It will carry K3s, both Bifrost releases with their Postgres, the staging and production app databases, cloudflared, the runner and niobi-proxy, so memory is the number to watch (P0.08).
App databases: Postgres 17 for staging, production and CI (decisions log, Ledger database). Production data on the droplet’s disk, backed up to R2; no separate Volume. Installed in P0.08.
Cloudflare, both environments:
- Zone
router.africa: SSL/TLS Full (strict), Always Use HTTPS, Universal SSL active. Checked: the website andprod-bifrost-corestill load. - R2 buckets:
router-africa-content-staging,router-africa-content-production,router-africa-db-backups-production(all private). - R2 token
db-backups-production: Object Read & Write on the backup bucket only. - Queues:
usage-events-staging,usage-events-production,niobi-callbacks-staging,niobi-callbacks-production. - Zero Trust: team
routerafrica(routerafrica.cloudflareaccess.com), Free plan. Nothing configured yet. - Deploy API token: one token, this account and zone only.
- No app DNS records yet: the custom domains create them in P0.07. Workers Builds and the Pages Git integration are not connected.
Every token is in Bitwarden (../infra/secrets-inventory.md).
15. App databases and production Bifrost under tracked values (2026-11-01, P0.08)
Section titled “15. App databases and production Bifrost under tracked values (2026-11-01, P0.08)”Run by the operator from the code repo’s infra/README.md (code #57), as bifrost over SSH. Values files live in the code repo under infra/; nothing was typed by hand except secrets.
Measured before: node memory 2122 Mi used (54%), 1972 Mi available, no swap, disk 60 GB free.
Production Bifrost: helm diff upgrade against infra/gateways/bifrost-1/values-production.yaml (chart 2.1.43) showed no changes. helm upgrade then recorded it as revision 5; the pods weren’t restarted. The chart’s notice “PostgreSQL is using the default password bifrost_password” is a false alarm: both Bifrost and its Postgres read the password from the postgres-credentials secret, and the notice only checks the unused default value.
No staging Bifrost (decisions log, Daily backups; no staging Bifrost).
CloudNativePG: operator 1.30.1 (chart cnpg/cloudnative-pg 0.29.1) in cnpg-system.
App databases: release app-db (chart cnpg/cluster 0.9.0) in postgres-staging (5 Gi) and postgres-production (20 Gi), cluster app-db-cluster, Postgres 17.11 (ghcr.io/cloudnative-pg/postgresql:17.11-202610260819-standard-bookworm), one instance each, local-path storage on the droplet’s disk. Database router_africa, owner migrator. Both “Cluster in healthy state”; helm test passed on both. Read-write service: app-db-cluster-rw.<namespace>.svc:5432.
Secrets (created on the droplet; copies in Bitwarden):
postgres-superuser(basic-auth, userpostgres) in each namespace;r2-backup-credentials(ACCESS_KEY_ID,ACCESS_SECRET_KEY, from thedb-backups-productionR2 token) inpostgres-production.
Daily backup: release db-backup (infra/postgres/daily-backup), a CronJob at 03:00 UTC. It runs pg_dump --format=custom and uploads to router-africa-db-backups-production/production/daily/<UTC timestamp>.dump, via R2’s S3 endpoint for this account. The first run uploaded production/daily/2026-11-01T2050Z.dump. Kept 30 days by an R2 lifecycle rule on production/daily/. Restore steps: infra/README.md, Restoring.
Measured after: 1693 Mi available. The operator uses 31 Mi and each database about 60 Mi idle.
Pending: the droplet reports “System restart required” (Ubuntu updates). Plan a short reboot after setup; every pod restarts with it.
Hardening applied (2026-11-05): the reviewer hardened the chart after this install (code #57, ce9457646): the daily backup’s containers and niobi-proxy run with no Linux capabilities, no privilege escalation and the default seccomp profile. helm upgrade db-backup … --reuse-values (revision 2) applied it; --reuse-values keeps the R2 endpointURL set at install.
16. Tunnel, gateway route split and Workers VPC Services (2026-11-01, P0.10)
Section titled “16. Tunnel, gateway route split and Workers VPC Services (2026-11-01, P0.10)”From the code repo’s infra/README.md, P0.10 (code #60).
Tunnel: router-africa-cluster, ID 5568057a-6a6a-4ac7-91fa-075084a0abc4, created under Workers & Pages → VPC → Tunnels (remote-managed). It runs as release cloudflared (infra/cloudflared, image cloudflare/cloudflared:2026.10.2) in namespace cloudflared, 2 replicas, outbound only. Its token is the cloudflared-token secret, with a copy in Bitwarden.
Route split: release route-split (infra/gateways/bifrost-1/route-split, nginxinc/nginx-unprivileged:1.31.7-alpine) in default, in front of production Bifrost. It runs nginx rather than extra Traefik entrypoints, because Traefik also serves prod-bifrost-core..
- Service
bifrost-inference:POST /v1/chat/completionsandGET /healthonly. - Service
bifrost-management:GET|POST /api/governance/virtual-keys,GET|PUT|DELETE /api/governance/virtual-keys/{id},GET /api/models,GET /api/models/details(added 2026-11-05, code #70) andGET /api/logsonly. - Checked in the cluster: inference
/healthgives 200; management/api/providersgives 404. - Upgraded 2026-11-05 (
route-splitandcloudflaredat revision 2), applying three changes merged after the install: (1) the reviewer’s fix (code #60,1334d9c74): nginx checked the allow-list against the normalised path but forwarded the raw request line, so a path that normalises to an allowed route could reach Bifrost as a different one; each route now forwards exactly what it matched; (2)GET /api/models/detailson management (code #70, for the P3.01 adapter’slistModels); (3) no Linux capabilities and the default seccomp profile forroute-splitandcloudflared. Checked in the cluster after: inference/health200; management/api/providers404;/api/models/details401 (reaches Bifrost, no admin credential sent);/api/governance/virtual-keys/..%2F..%2Fproviders404; twocloudflaredpods Running. - Copying
infra/to the droplet:scp -r infra bifrost@…:~/router-africa-infracopies into~/router-africa-infra/infra/when that folder already exists, leaving the old files in place (it happened on 2026-11-05:helm diffshowed nothing). Move the old folder aside first, or copyinfra/.into it.
Workers VPC Services (all on this tunnel):
| Name | ID | Target | Used by |
|---|---|---|---|
gateway-inference-production |
01a19908-6838-7943-b8b5-99e6496e8985 |
bifrost-inference.default.svc.cluster.local:80 (http) |
apps/api production, binding GATEWAY_INFERENCE |
gateway-management-production |
01a19908-718c-79b0-b75e-889fbfd540d0 |
bifrost-management.default.svc.cluster.local:80 (http) |
apps/dashboard-api production, binding GATEWAY_MANAGEMENT |
postgres-production |
01a19908-7b12-77a1-926c-03ca79b4174c |
app-db-cluster-rw.postgres-production.svc.cluster.local:5432 (tcp, postgresql) |
Hyperdrive, production (P1.04/P1.06) |
postgres-staging |
01a19908-8432-71a2-b862-109ab881a83c |
app-db-cluster-rw.postgres-staging.svc.cluster.local:5432 (tcp, postgresql) |
Hyperdrive, staging (P1.04/P1.06) |
The Postgres services have certificate verification disabled: CloudNativePG uses a private CA, and Workers VPC can’t take a custom CA yet. Traffic is encrypted by the tunnel. The follow-up is a Cloudflare Origin CA certificate.
Not yet checked: a call from a deployed Worker. That waits for the Hyperdrive configs (P1.04/P1.06) and the Bifrost adapter (P3.01).
17. Database roles, the staging schema and Hyperdrive (2026-11-01, P1.06)
Section titled “17. Database roles, the staging schema and Hyperdrive (2026-11-01, P1.06)”From the code repo’s infra/README.md, P1.06 (code #62, #63).
Roles: api, dashboard and consumer are CloudNativePG managed roles (cluster.roles) in both clusters, with LOGIN and passwords from the secrets db-role-<role> (copies in Bitwarden). The operator reports them reconciled. migrator owns the database, and its credentials are the operator’s app-db-cluster-app secret.
Staging schema: first applied 2026-11-01 (migrations 0000–0009). The stack’s migrations were then renumbered, so on 2026-11-02 the core, logs and drizzle schemas were dropped and the migrations applied again as migrator from api.p3.01-bifrost-adapter (now all in main), through an SSH-forwarded port-forward. Checked in the database:
- 12 migrations recorded;
- the grants work:
apican insert intocore.holdsand can’t readcore.users.
Port 15432 on the droplet was held by something else (not a port-forward), so the tunnel used droplet port 25432: ssh -L 15432:localhost:25432 … port-forward svc/app-db-cluster-rw 25432:5432. The local port and DATABASE_URL stay the same.
Production schema: not applied yet. It waits for the merged code and the pipeline (P0.09).
Hyperdrive configs: each was created through the postgres-* VPC Service, which also proved the path tunnel → database → login.
| Config | ID | Role | Bound as |
|---|---|---|---|
router-africa-api-staging |
15cc51f82c884663b689819de26cd3f1 |
api |
apps/api HYPERDRIVE |
router-africa-dashboard-staging |
9677438c4d2846ccb0a32bbefd1188f0 |
dashboard |
apps/dashboard-api HYPERDRIVE |
router-africa-consumer-staging |
7993e3255d0e4cd4a76cefd4ee477698 |
consumer |
apps/dashboard-api HYPERDRIVE_LOGS |
router-africa-api-production |
5f8de627f5fb4319bc0421a4ef3b3619 |
api |
apps/api HYPERDRIVE |
router-africa-dashboard-production |
8739deb25ac5405c89c624905a6a57fc |
dashboard |
apps/dashboard-api HYPERDRIVE |
router-africa-consumer-production |
49e7e5e6a54c4aec8677c1bd2197b52f |
consumer |
apps/dashboard-api HYPERDRIVE_LOGS |
Query caching is off in all six (since 2026-11-05). They were created with caching on: the runbook’s create commands lacked --caching-disabled until the reviewer’s fix (code #63, fda3fda02), and the check on 2026-11-03 found caching.disabled: false in all six. The decisions log requires it off: a cached read could serve a stale balance, or keep a revoked key working for up to 60 seconds. Turned off with npx wrangler hyperdrive update <id> --caching-disabled; checked with wrangler hyperdrive get: caching.disabled: true in all six.
Remaining work (not yet done)
Section titled “Remaining work (not yet done)”- Provider API keys — OpenAI, Anthropic, Google, Fireworks keys not yet available. Once obtained: either add via the dashboard directly, or create a
provider-keyssecret and add aproviders:/env:block tovalues.yaml, thenhelm upgrade bifrost bifrost/bifrost -f values.yaml. Dashboard password protection— done. Password protection enabled on the dashboard.- Backups (Bifrost’s own Postgres) — no backup job configured yet for Bifrost’s Postgres data (the app databases have theirs, section 15). Set up a
pg_dumpcron job (e.g. a KubernetesCronJobdumping to an off-cluster destination, such as DigitalOcean Spaces) once there’s real data worth protecting. Deferred through the v2.2.3 upgrade since the instance held only an admin account at that point — revisit before onboarding real usage. End-to-end inference test— skipped for now; blocked on provider keys (item 1). Revisit once those are added.- Rancher — deliberately deferred. See
rancher-setup-TODO.md. Saved secrets— done. Encryption key and Postgres password both retrieved from their Kubernetes secrets and saved outside the server.