Skip to content
Router.Africa Context

Router.Africa — Bifrost Installation (As-Built)

Status: Live in production Server: [redacted] — DigitalOcean Droplet, [redacted-ip], Ubuntu 26.04 LTS Gateway URL: https://prod-bifrost-core.router.africa Last updated: 2026-09-14

This documents what was actually installed and configured on this server, in order, so it can be reproduced or audited later. Rancher was deliberately not used — see rancher-setup-TODO.md for that deferred piece.

1. Server rebuild

Section titled “1. Server rebuild”

The Droplet was rebuilt clean via the DigitalOcean control panel (Droplet page → Destroy tab → Rebuild) on image Ubuntu 26.04 LTS (“Resolute Raccoon”), keeping the same IP ([redacted-ip]).

2. Initial user setup

Section titled “2. Initial user setup”

Logged in as root using the one-time password DigitalOcean emailed after the rebuild, then:

Terminal window
apt update && apt upgrade -y
adduser bifrost
usermod -aG sudo bifrost

SSH key-based access was set up for bifrost via ssh-copy-id from the local machine (rather than copying root’s .ssh, since root had no key on it), and confirmed working with sudo whoami returning root.

3. Firewall

Section titled “3. Firewall”
Terminal window
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Port 6443 (Kubernetes API) is deliberately not opened to the internet. An IP-restricted rule was tried first, then removed once it became clear the operator isn’t on a static IP — kubectl is only used locally on the Droplet itself via SSH, so 6443 doesn’t need to be reachable externally at all. Current state, confirmed via sudo ufw status:

OpenSSH                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
(+ IPv6 equivalents)

4. K3s

Section titled “4. K3s”
Terminal window
curl -sfL https://get.k3s.io | sh -

Installed as v1.36.4+k3s1, single control-plane node, ships with Traefik (ingress) and ServiceLB built in.

kubectl configured for the bifrost user (not root, not sudo k3s kubectl):

Terminal window
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc

5. Helm

Section titled “5. Helm”
Terminal window
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Installed: v3.22.0.

6. cert-manager

Section titled “6. cert-manager”
Terminal window
helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true

ClusterIssuer for Let’s Encrypt:

Terminal window
cat > cluster-issuer.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: mesyedhassan@gmail.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: traefik
EOF


kubectl apply -f cluster-issuer.yaml

7. DNS

Section titled “7. DNS”

A record added at the DNS provider managing router.africa (proxy/orange-cloud left off — required for the Let’s Encrypt HTTP-01 challenge to reach the Droplet directly):

Type: A
Name: prod-bifrost-core
Value: [redacted-ip]

8. Secrets

Section titled “8. Secrets”
Terminal window
ENCRYPTION_KEY=$(openssl rand -base64 32)
kubectl create secret generic bifrost-encryption-key \
  --from-literal=encryption-key="$ENCRYPTION_KEY"
# (value saved outside the cluster separately)


kubectl create secret generic postgres-credentials \
  --from-literal=password="$(openssl rand -base64 24)"

Provider API keys (provider-keys secret) were not created yet — keys weren’t available at install time. See “Remaining work” below.

9. Bifrost Helm repo

Section titled “9. Bifrost Helm repo”
Terminal window
helm repo add bifrost https://maximhq.github.io/bifrost/helm-charts
helm repo update

Installed chart: bifrost-2.1.40, app version 1.5.12.

10. Helm values (final, working version)

Section titled “10. Helm values (final, working version)”

This is the validated values.yaml actually deployed — reached after correcting two issues the first draft got wrong against the real chart schema (see “Issues hit during install” below):

image:
  tag: "v1.5.12"


replicaCount: 1


storage:
  mode: postgres
  persistence:
    enabled: false
  configStore:
    enabled: true
  logsStore:
    enabled: true


postgresql:
  enabled: true
  auth:
    username: bifrost
    database: bifrost
    existingSecret: "postgres-credentials"
    passwordKey: "password"
  primary:
    persistence:
      enabled: true
      size: 8Gi
    resources:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 1Gi


resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi


livenessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 45
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6


readinessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 45
  periodSeconds: 5
  timeoutSeconds: 5
  failureThreshold: 6


ingress:
  enabled: true
  className: traefik
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: prod-bifrost-core.router.africa
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: bifrost-tls
      hosts:
        - prod-bifrost-core.router.africa


bifrost:
  encryptionKeySecret:
    name: "bifrost-encryption-key"
    key: "encryption-key"
  client:
    allowedOrigins:
      - "https://router.africa"
      - "https://prod-bifrost-core.router.africa"
    enforceAuthOnInference: true


env:
  - name: GOGC
    value: "200"
  - name: GOMEMLIMIT
    value: "460MiB"

Install command:

Terminal window
helm install bifrost bifrost/bifrost -f values.yaml

11. Issues hit during install (and fixes)

Section titled “11. Issues hit during install (and fixes)”

Schema mismatch on first attempt. helm install --dry-run caught this immediately:

Error: at '/bifrost/client': additional properties 'enforce_auth_on_inference', 'allowed_origins' not allowed

Fix: the chart uses camelCase (enforceAuthOnInference, allowedOrigins), not the snake_case used in raw config.json. Also, postgresql is a top-level values key in this chart, not nested under storage.postgresql as first assumed — the nested version wouldn’t have errored, it would have silently deployed nothing.

Missing required field. image.tag has no default and must be set explicitly — resolved by checking helm search repo bifrost/bifrost --versions for the current app version (1.5.12) and using tag v1.5.12.

CrashLoopBackOff after a clean dry-run and install. The pod alternated between Running and Error/CrashLoopBackOff. kubectl logs <pod> --previous showed the real cause: Bifrost logged "Time spent in Bifrost server bootstrap 33805 ms" and "successfully started", then 14 seconds later received SIGTERM and shut down gracefully — it wasn’t crashing internally, Kubernetes was killing it. The chart’s default livenessProbe (5s initial delay + 10s period × 3 failures = 35s total) was too tight against a ~34s real startup time (Postgres connection, config/logs store migrations, model catalog sync, MCP library sync).

First fix attempt (startupProbe) didn’t work — it wasn’t rejected by schema validation, but kubectl describe deployment confirmed the chart template doesn’t render a startupProbe field at all, so it was silently ignored. Actual fix: raised livenessProbe/readinessProbe initialDelaySeconds to 45 and failureThreshold to 6 (giving ~105s of margin) — these probes back the pod’s real health checks and cover it correctly, confirmed via kubectl describe deployment bifrost showing delay=45s ... #failure=6 on both.

12. Verification

Section titled “12. Verification”
Terminal window
kubectl get pods
# bifrost-56df86bcb-ll2ln    1/1  Running  0 restarts
# bifrost-postgresql-...     1/1  Running  0 restarts


kubectl get certificate
# bifrost-tls   READY: True


curl https://prod-bifrost-core.router.africa/health
# {"components":{"db_pings":"ok"},"status":"ok"}

Full chain confirmed working end to end: DNS → Let’s Encrypt/HTTPS → Traefik ingress → Bifrost → self-hosted Postgres.

Remaining work (not yet done)

Section titled “Remaining work (not yet done)”
  1. Provider API keys — OpenAI, Anthropic, Google, Fireworks keys not yet available. Once obtained: either add via the dashboard directly, or create a provider-keys secret and add a providers:/env: block to values.yaml, then helm upgrade bifrost bifrost/bifrost -f values.yaml.
  2. Dashboard password protection — done. Password protection enabled on the dashboard.
  3. Backups — no backup job configured yet for the Postgres data. Set up the pg_dump cron job (documented in bifrost-rancher-k8s-deployment-guide.md, Section 12) once there’s real data worth protecting.
  4. End-to-end inference test — skipped for now; blocked on provider keys (item 1). Revisit once those are added.
  5. Rancher — deliberately deferred. See rancher-setup-TODO.md.
  6. Saved secrets — done. Encryption key and Postgres password both retrieved from their Kubernetes secrets and saved outside the server.