Skip to content

Router.Africa — Bifrost Installation (As-Built)

Status: Live in production Server: [redacted] — DigitalOcean Droplet, [redacted-ip], Ubuntu 26.04 LTS Gateway URL: https://prod-bifrost-core.router.africa Bifrost version: v2.2.3 (chart bifrost-2.1.43) Last updated: 2026-11-01 (P0.08 at 20:50 UTC)

Planned changes before public launch (decided 2026-10-25, not yet applied). This file stays the record of what is installed; these change it at the pre-launch lockdown (decisions log, Droplet network lockdown, Private service access, Niobi webhooks, Gateway abstraction):

  • Firewall: move from ufw to the DigitalOcean Cloud Firewall (K3s’s iptables rules can bypass ufw). Inbound closed except the WireGuard UDP port and 443 from Niobi’s webhook IPs. SSH, 80 and public 443 closed. Test the DigitalOcean web console as the recovery path first.
  • Operator access: WireGuard for SSH, kubectl and the Bifrost dashboard.
  • Bifrost: remove the prod-bifrost-core.router.africa ingress and DNS record. Workers reach Bifrost through cloudflared (in-cluster, outbound only) and a Workers VPC Service. The Workers use that path from day one; the public hostname is for the operator during the build only.
  • Layout: Helm config moves to infra/gateways/bifrost-1/ in the code repo; pin the exact Bifrost image and keep a copy.
  • cert-manager: the Niobi webhook host uses the DNS-01 challenge (Cloudflare API token), since port 80 will be closed.

This documents what was actually installed and configured on this server, in order, so it can be reproduced or audited later. Rancher was deliberately not used — see rancher-setup-TODO.md for that deferred piece.


The Droplet was rebuilt clean via the DigitalOcean control panel (Droplet page → Destroy tab → Rebuild) on image Ubuntu 26.04 LTS (“Resolute Raccoon”), keeping the same IP ([redacted-ip]).

Logged in as root using the one-time password DigitalOcean emailed after the rebuild, then:

Terminal window
apt update && apt upgrade -y
adduser bifrost
usermod -aG sudo bifrost

SSH key-based access was set up for bifrost via ssh-copy-id from the local machine (rather than copying root’s .ssh, since root had no key on it), and confirmed working with sudo whoami returning root.

Terminal window
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Port 6443 (Kubernetes API) is deliberately not opened to the internet. An IP-restricted rule was tried first, then removed once it became clear the operator isn’t on a static IP — kubectl is only used locally on the Droplet itself via SSH, so 6443 doesn’t need to be reachable externally at all. Current state, confirmed via sudo ufw status:

OpenSSH ALLOW Anywhere
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
(+ IPv6 equivalents)
Terminal window
curl -sfL https://get.k3s.io | sh -

Installed as v1.36.4+k3s1, single control-plane node, ships with Traefik (ingress) and ServiceLB built in.

kubectl configured for the bifrost user (not root, not sudo k3s kubectl):

Terminal window
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
Terminal window
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Installed: v3.22.0.

Terminal window
helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true

ClusterIssuer for Let’s Encrypt:

Terminal window
cat > cluster-issuer.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: mesyedhassan@gmail.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: traefik
EOF
kubectl apply -f cluster-issuer.yaml

A record added at the DNS provider managing router.africa (proxy/orange-cloud left off — required for the Let’s Encrypt HTTP-01 challenge to reach the Droplet directly):

Type: A
Name: prod-bifrost-core
Value: [redacted-ip]
Terminal window
ENCRYPTION_KEY=$(openssl rand -base64 32)
kubectl create secret generic bifrost-encryption-key \
--from-literal=encryption-key="$ENCRYPTION_KEY"
# (value saved outside the cluster separately)
kubectl create secret generic postgres-credentials \
--from-literal=password="$(openssl rand -base64 24)"

Provider API keys (provider-keys secret) were not created yet — keys weren’t available at install time. See “Remaining work” below.

Terminal window
helm repo add bifrost https://maximhq.github.io/bifrost/helm-charts
helm repo update

Installed chart: bifrost-2.1.40, app version 1.5.12. (Upgraded to chart 2.1.43 / app v2.2.3 on 2026-09-27 — see Section 13.)

This is the validated values.yaml actually deployed — reached after correcting two issues the first draft got wrong against the real chart schema (see “Issues hit during install” below):

image:
tag: "v2.2.3" # originally v1.5.12 at install; upgraded 2026-09-27, see Section 13
replicaCount: 1
storage:
mode: postgres
persistence:
enabled: false
configStore:
enabled: true
logsStore:
enabled: true
postgresql:
enabled: true
auth:
username: bifrost
database: bifrost
existingSecret: "postgres-credentials"
passwordKey: "password"
primary:
persistence:
enabled: true
size: 8Gi
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 45
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 45
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 6
ingress:
enabled: true
className: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: prod-bifrost-core.router.africa
paths:
- path: /
pathType: Prefix
tls:
- secretName: bifrost-tls
hosts:
- prod-bifrost-core.router.africa
bifrost:
encryptionKeySecret:
name: "bifrost-encryption-key"
key: "encryption-key"
client:
allowedOrigins:
- "https://router.africa"
- "https://prod-bifrost-core.router.africa"
enforceAuthOnInference: true
env:
- name: GOGC
value: "200"
- name: GOMEMLIMIT
value: "460MiB"

Install command:

Terminal window
helm install bifrost bifrost/bifrost -f values.yaml

Schema mismatch on first attempt. helm install --dry-run caught this immediately:

Error: at '/bifrost/client': additional properties 'enforce_auth_on_inference', 'allowed_origins' not allowed

Fix: the chart uses camelCase (enforceAuthOnInference, allowedOrigins), not the snake_case used in raw config.json. Also, postgresql is a top-level values key in this chart, not nested under storage.postgresql as first assumed — the nested version wouldn’t have errored, it would have silently deployed nothing.

Missing required field. image.tag has no default and must be set explicitly — resolved by checking helm search repo bifrost/bifrost --versions for the current app version (1.5.12) and using tag v1.5.12.

CrashLoopBackOff after a clean dry-run and install. The pod alternated between Running and Error/CrashLoopBackOff. kubectl logs <pod> --previous showed the real cause: Bifrost logged "Time spent in Bifrost server bootstrap 33805 ms" and "successfully started", then 14 seconds later received SIGTERM and shut down gracefully — it wasn’t crashing internally, Kubernetes was killing it. The chart’s default livenessProbe (5s initial delay + 10s period × 3 failures = 35s total) was too tight against a ~34s real startup time (Postgres connection, config/logs store migrations, model catalog sync, MCP library sync).

First fix attempt (startupProbe) didn’t work — it wasn’t rejected by schema validation, but kubectl describe deployment confirmed the chart template doesn’t render a startupProbe field at all, so it was silently ignored. Actual fix: raised livenessProbe/readinessProbe initialDelaySeconds to 45 and failureThreshold to 6 (giving ~105s of margin) — these probes back the pod’s real health checks and cover it correctly, confirmed via kubectl describe deployment bifrost showing delay=45s ... #failure=6 on both.

12. Upgrade: v1.5.12 → v2.2.3 (2026-09-27)

Section titled “12. Upgrade: v1.5.12 → v2.2.3 (2026-09-27)”

The instance had been running stable for ~12 days on v1.5.12 with no data beyond an admin account, so this upgrade was done without a Postgres backup first — an explicit call given the low stakes (fresh instance, nothing valuable to lose yet). This is not the standard going forward once real data exists; see “Remaining work” item 3 below.

Step 1 — refresh Helm repo as the right user. helm repo update failed under root (Error: no repositories found. You must add one before updating) because the bifrost Helm repo was only ever added under the bifrost user, not root. Fixed by switching user first:

Terminal window
su - bifrost
helm repo update
helm search repo bifrost/bifrost --versions

This listed chart versions 2.1.34 through 2.1.43, but all of them showed app version 1.5.12 in helm search output — the chart’s Chart.yaml appVersion field hadn’t been bumped in lockstep with actual Bifrost releases, so chart version was not a reliable proxy for the image tag.

Step 2 — verify the target image tag actually exists. Queried Docker Hub’s v2 API directly rather than trusting helm search:

Terminal window
curl -s "https://hub.docker.com/v2/repositories/maximhq/bifrost/tags?page_size=100" | grep -o '"name":"[^"]*"'

A first pass returned tags in an unhelpful default order that made it look like v2.2.3 might not exist; re-running with explicit sorting/filtering on the tag name confirmed v2.2.3 is a real, published tag.

Step 3 — update values.yaml and dry-run.

Terminal window
sed -i 's/tag: "v1.5.12"/tag: "v2.2.3"/' values.yaml
helm upgrade bifrost bifrost/bifrost -f values.yaml --dry-run

Dry-run output confirmed: chart moves from 2.1.40 → 2.1.43, image tag from v1.5.12 → v2.2.3, no other schema changes needed — the values.yaml structure from the original install (Section 10) was still valid against the new chart version.

Step 4 — real upgrade.

Terminal window
helm upgrade bifrost bifrost/bifrost -f values.yaml
kubectl get pods -w

Issue hit: transient Postgres connection-refused on the new Bifrost pod. The new bifrost pod came up with 2 restarts. kubectl logs <pod> --previous showed:

failed to connect to `user=bifrost database=bifrost`: 10.43.139.154:5432 (bifrost-postgresql): dial error: dial tcp 10.43.139.154:5432: connect: connection refused

Diagnosis: the bundled Postgres sub-chart’s Deployment uses strategy: Recreate (not a rolling update), so helm upgrade tore down and recreated the Postgres pod at roughly the same time as the Bifrost pod. Bifrost’s new pod started faster than Postgres came back up, tried to connect, failed, and was restarted by Kubernetes — which then succeeded once Postgres was ready. Not a v2.2.3 regression, just an ordering race between two pods with no dependency/readiness gate between them at the Helm level.

Confirmed resolved (not an ongoing crash loop) by watching restart count hold steady rather than climb, then checking health:

Terminal window
kubectl get pods
# bifrost-d49cdf95f-fcwxj 1/1 Running 2 (2m59s ago) 3m11s
# bifrost-postgresql-5b7b668d6f-27976 1/1 Running 0 3m10s
curl https://prod-bifrost-core.router.africa/health
# {"components":{"db_pings":"ok"},"status":"ok"}

Takeaway for future upgrades: the Postgres-recreate race is expected behavior with this chart’s current settings, not a fault — a brief restart or two on the Bifrost pod right after an upgrade is normal and self-resolves once Postgres is back. If this needs to be eliminated (e.g. once there’s real traffic and even a few seconds of restart noise matters), options to revisit: pin the Postgres sub-chart to strategy: RollingUpdate if the chart exposes that, or add an init container / readiness gate on Bifrost that waits for Postgres before starting.

Terminal window
kubectl get pods
# bifrost-56df86bcb-ll2ln 1/1 Running 0 restarts
# bifrost-postgresql-... 1/1 Running 0 restarts
kubectl get certificate
# bifrost-tls READY: True
curl https://prod-bifrost-core.router.africa/health
# {"components":{"db_pings":"ok"},"status":"ok"}

Full chain confirmed working end to end: DNS → Let’s Encrypt/HTTPS → Traefik ingress → Bifrost → self-hosted Postgres.


14. Platform setup for the app (2026-11-01, P0.02 and P0.03)

Section titled “14. Platform setup for the app (2026-11-01, P0.02 and P0.03)”

Droplet capacity: 2 vCPU, 4 GB RAM, 70 GB disk. It will carry K3s, both Bifrost releases with their Postgres, the staging and production app databases, cloudflared, the runner and niobi-proxy, so memory is the number to watch (P0.08).

App databases: Postgres 17 for staging, production and CI (decisions log, Ledger database). Production data on the droplet’s disk, backed up to R2; no separate Volume. Installed in P0.08.

Cloudflare, both environments:

  • Zone router.africa: SSL/TLS Full (strict), Always Use HTTPS, Universal SSL active. Checked: the website and prod-bifrost-core still load.
  • R2 buckets: router-africa-content-staging, router-africa-content-production, router-africa-db-backups-production (all private).
  • R2 token db-backups-production: Object Read & Write on the backup bucket only.
  • Queues: usage-events-staging, usage-events-production, niobi-callbacks-staging, niobi-callbacks-production.
  • Zero Trust: team routerafrica (routerafrica.cloudflareaccess.com), Free plan. Nothing configured yet.
  • Deploy API token: one token, this account and zone only.
  • No app DNS records yet: the custom domains create them in P0.07. Workers Builds and the Pages Git integration are not connected.

Every token is in Bitwarden (../infra/secrets-inventory.md).

15. App databases and production Bifrost under tracked values (2026-11-01, P0.08)

Section titled “15. App databases and production Bifrost under tracked values (2026-11-01, P0.08)”

Run by the operator from the code repo’s infra/README.md (code #57), as bifrost over SSH. Values files live in the code repo under infra/; nothing was typed by hand except secrets.

Measured before: node memory 2122 Mi used (54%), 1972 Mi available, no swap, disk 60 GB free.

Production Bifrost: helm diff upgrade against infra/gateways/bifrost-1/values-production.yaml (chart 2.1.43) showed no changes. helm upgrade then recorded it as revision 5; the pods weren’t restarted. The chart’s notice “PostgreSQL is using the default password bifrost_password” is a false alarm: both Bifrost and its Postgres read the password from the postgres-credentials secret, and the notice only checks the unused default value.

No staging Bifrost (decisions log, Daily backups; no staging Bifrost).

CloudNativePG: operator 1.30.1 (chart cnpg/cloudnative-pg 0.29.1) in cnpg-system.

App databases: release app-db (chart cnpg/cluster 0.9.0) in postgres-staging (5 Gi) and postgres-production (20 Gi), cluster app-db-cluster, Postgres 17.11 (ghcr.io/cloudnative-pg/postgresql:17.11-202610260819-standard-bookworm), one instance each, local-path storage on the droplet’s disk. Database router_africa, owner migrator. Both “Cluster in healthy state”; helm test passed on both. Read-write service: app-db-cluster-rw.<namespace>.svc:5432.

Secrets (created on the droplet; copies in Bitwarden):

  • postgres-superuser (basic-auth, user postgres) in each namespace;
  • r2-backup-credentials (ACCESS_KEY_ID, ACCESS_SECRET_KEY, from the db-backups-production R2 token) in postgres-production.

Daily backup: release db-backup (infra/postgres/daily-backup), a CronJob at 03:00 UTC. It runs pg_dump --format=custom and uploads to router-africa-db-backups-production/production/daily/<UTC timestamp>.dump, via R2’s S3 endpoint for this account. The first run uploaded production/daily/2026-11-01T2050Z.dump. Kept 30 days by an R2 lifecycle rule on production/daily/. Restore steps: infra/README.md, Restoring.

Measured after: 1693 Mi available. The operator uses 31 Mi and each database about 60 Mi idle.

Pending: the droplet reports “System restart required” (Ubuntu updates). Plan a short reboot after setup; every pod restarts with it.

Hardening applied (2026-11-05): the reviewer hardened the chart after this install (code #57, ce9457646): the daily backup’s containers and niobi-proxy run with no Linux capabilities, no privilege escalation and the default seccomp profile. helm upgrade db-backup … --reuse-values (revision 2) applied it; --reuse-values keeps the R2 endpointURL set at install.

16. Tunnel, gateway route split and Workers VPC Services (2026-11-01, P0.10)

Section titled “16. Tunnel, gateway route split and Workers VPC Services (2026-11-01, P0.10)”

From the code repo’s infra/README.md, P0.10 (code #60).

Tunnel: router-africa-cluster, ID 5568057a-6a6a-4ac7-91fa-075084a0abc4, created under Workers & Pages → VPC → Tunnels (remote-managed). It runs as release cloudflared (infra/cloudflared, image cloudflare/cloudflared:2026.10.2) in namespace cloudflared, 2 replicas, outbound only. Its token is the cloudflared-token secret, with a copy in Bitwarden.

Route split: release route-split (infra/gateways/bifrost-1/route-split, nginxinc/nginx-unprivileged:1.31.7-alpine) in default, in front of production Bifrost. It runs nginx rather than extra Traefik entrypoints, because Traefik also serves prod-bifrost-core..

  • Service bifrost-inference: POST /v1/chat/completions and GET /health only.
  • Service bifrost-management: GET|POST /api/governance/virtual-keys, GET|PUT|DELETE /api/governance/virtual-keys/{id}, GET /api/models, GET /api/models/details (added 2026-11-05, code #70) and GET /api/logs only.
  • Checked in the cluster: inference /health gives 200; management /api/providers gives 404.
  • Upgraded 2026-11-05 (route-split and cloudflared at revision 2), applying three changes merged after the install: (1) the reviewer’s fix (code #60, 1334d9c74): nginx checked the allow-list against the normalised path but forwarded the raw request line, so a path that normalises to an allowed route could reach Bifrost as a different one; each route now forwards exactly what it matched; (2) GET /api/models/details on management (code #70, for the P3.01 adapter’s listModels); (3) no Linux capabilities and the default seccomp profile for route-split and cloudflared. Checked in the cluster after: inference /health 200; management /api/providers 404; /api/models/details 401 (reaches Bifrost, no admin credential sent); /api/governance/virtual-keys/..%2F..%2Fproviders 404; two cloudflared pods Running.
  • Copying infra/ to the droplet: scp -r infra bifrost@…:~/router-africa-infra copies into ~/router-africa-infra/infra/ when that folder already exists, leaving the old files in place (it happened on 2026-11-05: helm diff showed nothing). Move the old folder aside first, or copy infra/. into it.

Workers VPC Services (all on this tunnel):

Name ID Target Used by
gateway-inference-production 01a19908-6838-7943-b8b5-99e6496e8985 bifrost-inference.default.svc.cluster.local:80 (http) apps/api production, binding GATEWAY_INFERENCE
gateway-management-production 01a19908-718c-79b0-b75e-889fbfd540d0 bifrost-management.default.svc.cluster.local:80 (http) apps/dashboard-api production, binding GATEWAY_MANAGEMENT
postgres-production 01a19908-7b12-77a1-926c-03ca79b4174c app-db-cluster-rw.postgres-production.svc.cluster.local:5432 (tcp, postgresql) Hyperdrive, production (P1.04/P1.06)
postgres-staging 01a19908-8432-71a2-b862-109ab881a83c app-db-cluster-rw.postgres-staging.svc.cluster.local:5432 (tcp, postgresql) Hyperdrive, staging (P1.04/P1.06)

The Postgres services have certificate verification disabled: CloudNativePG uses a private CA, and Workers VPC can’t take a custom CA yet. Traffic is encrypted by the tunnel. The follow-up is a Cloudflare Origin CA certificate.

Not yet checked: a call from a deployed Worker. That waits for the Hyperdrive configs (P1.04/P1.06) and the Bifrost adapter (P3.01).

17. Database roles, the staging schema and Hyperdrive (2026-11-01, P1.06)

Section titled “17. Database roles, the staging schema and Hyperdrive (2026-11-01, P1.06)”

From the code repo’s infra/README.md, P1.06 (code #62, #63).

Roles: api, dashboard and consumer are CloudNativePG managed roles (cluster.roles) in both clusters, with LOGIN and passwords from the secrets db-role-<role> (copies in Bitwarden). The operator reports them reconciled. migrator owns the database, and its credentials are the operator’s app-db-cluster-app secret.

Staging schema: first applied 2026-11-01 (migrations 0000–0009). The stack’s migrations were then renumbered, so on 2026-11-02 the core, logs and drizzle schemas were dropped and the migrations applied again as migrator from api.p3.01-bifrost-adapter (now all in main), through an SSH-forwarded port-forward. Checked in the database:

  • 12 migrations recorded;
  • the grants work: api can insert into core.holds and can’t read core.users.

Port 15432 on the droplet was held by something else (not a port-forward), so the tunnel used droplet port 25432: ssh -L 15432:localhost:25432 … port-forward svc/app-db-cluster-rw 25432:5432. The local port and DATABASE_URL stay the same.

Production schema: not applied yet. It waits for the merged code and the pipeline (P0.09).

Hyperdrive configs: each was created through the postgres-* VPC Service, which also proved the path tunnel → database → login.

Config ID Role Bound as
router-africa-api-staging 15cc51f82c884663b689819de26cd3f1 api apps/api HYPERDRIVE
router-africa-dashboard-staging 9677438c4d2846ccb0a32bbefd1188f0 dashboard apps/dashboard-api HYPERDRIVE
router-africa-consumer-staging 7993e3255d0e4cd4a76cefd4ee477698 consumer apps/dashboard-api HYPERDRIVE_LOGS
router-africa-api-production 5f8de627f5fb4319bc0421a4ef3b3619 api apps/api HYPERDRIVE
router-africa-dashboard-production 8739deb25ac5405c89c624905a6a57fc dashboard apps/dashboard-api HYPERDRIVE
router-africa-consumer-production 49e7e5e6a54c4aec8677c1bd2197b52f consumer apps/dashboard-api HYPERDRIVE_LOGS

Query caching is off in all six (since 2026-11-05). They were created with caching on: the runbook’s create commands lacked --caching-disabled until the reviewer’s fix (code #63, fda3fda02), and the check on 2026-11-03 found caching.disabled: false in all six. The decisions log requires it off: a cached read could serve a stale balance, or keep a revoked key working for up to 60 seconds. Turned off with npx wrangler hyperdrive update <id> --caching-disabled; checked with wrangler hyperdrive get: caching.disabled: true in all six.

  1. Provider API keys — OpenAI, Anthropic, Google, Fireworks keys not yet available. Once obtained: either add via the dashboard directly, or create a provider-keys secret and add a providers:/env: block to values.yaml, then helm upgrade bifrost bifrost/bifrost -f values.yaml.
  2. Dashboard password protection — done. Password protection enabled on the dashboard.
  3. Backups (Bifrost’s own Postgres) — no backup job configured yet for Bifrost’s Postgres data (the app databases have theirs, section 15). Set up a pg_dump cron job (e.g. a Kubernetes CronJob dumping to an off-cluster destination, such as DigitalOcean Spaces) once there’s real data worth protecting. Deferred through the v2.2.3 upgrade since the instance held only an admin account at that point — revisit before onboarding real usage.
  4. End-to-end inference test — skipped for now; blocked on provider keys (item 1). Revisit once those are added.
  5. Rancher — deliberately deferred. See rancher-setup-TODO.md.
  6. Saved secrets — done. Encryption key and Postgres password both retrieved from their Kubernetes secrets and saved outside the server.