Skip to content

Router.Africa — Architecture Decisions Log

A running record of what’s decided versus still open. Update this as decisions get made — an item sitting in “open” indefinitely usually means it’s been silently deprioritized rather than resolved.


Decision Detail
Model gateway Bifrost, unchanged, deployed on the existing K3s droplet. Not rebuilt or replaced.
Ingest/auth/orchestration layer Single Node/TypeScript service, deployed as Cloudflare Workers. Chosen over Go and Python — at target throughput (100–500 req/s) raw performance isn’t the deciding factor; team fluency and type-sharing with the frontend were.
No Go in the main app Considered and rejected splitting request handling across Node (auth/balance) and Go (business logic/analytics/Bifrost call). Rejected because it introduces a distributed-transaction problem on the hold/settle path with no offsetting performance benefit at this scale — the workload is I/O-bound, not CPU-bound.
Bifrost call stays synchronous Standard chat completion requests are never queued. Clients expect real-time response, including streaming (SSE) — a queue is the wrong shape for that. Flow: Worker → hold → call Bifrost directly → settle → return.
What is queued Only post-response work: usage events, analytics, audit log. Fire-and-forget to Cloudflare Queues, consumed via a queue() handler inside apps/dashboard-api (not a separate Worker — see Queue consumer placement below). The client never waits on this.
Batch/async endpoint Explicitly out of scope for now. Standard request path only. Can be revisited later as a separate product surface (similar to OpenAI’s Batch API), not retrofitted onto the standard path.
Credit hold/settle mechanism Two-phase: hold before calling Bifrost, settle after. Router.Africa does not count input tokens. Hold = min(available balance, effective_max_tokens × output_rate + input_allowance). effective_max_tokens is the client’s max_tokens capped at a per-model ceiling, or a per-model default when the client omits it (never the model’s full output limit, which would over-reserve massively on large models). The per-model ceiling, default and input_allowance are configured in admin. A request is rejected only if available balance is below a configurable minimum hold. Settle: charge actual usage as reported by Bifrost at the locked rate and release the unused hold. If actual exceeds the hold, charge the full actual amount and floor the balance at zero, absorbing the shortfall. This is an accepted risk: occasional slight overage on the last request, in exchange for a simple hold; the next request is blocked. Monitor absorbed shortfalls and tune the input allowance.
Per-organization hold toggle Hold is switchable per organization in the admin app (revised 2026-09-20; earlier per account, then per workspace). It applies to every workspace in the org, and the request path reads it through the workspace’s org, so we can test which accounts actually need one. On (default): the full hold/settle flow above. Off: no hold is placed. The request is allowed if available balance is above the minimum hold threshold, and settle charges actual usage (Bifrost-reported tokens × the published rate in effect when the request started) with the balance floored at zero. Unheld concurrent requests can each pass the balance check, so overage is bounded only by concurrency and request size; the shortfall is absorbed, same as the accepted-risk rule above. The idempotency insert still runs. Toggle changes are audit-logged, and absorbed shortfalls are tracked per account so we can see whether holds-off accounts cost us anything.
Hold/settle concurrency safety Atomic conditional UPDATE in Postgres (WHERE balance - held >= :min_hold, setting the hold to LEAST(balance - held, :computed_hold)), not a Durable Object. A DO was originally proposed to solve the race condition, but once hold/settle are themselves durable DB writes, the DO adds a network hop without adding a concurrency guarantee Postgres doesn’t already provide row-locally.
Ledger database Revised 2026-11-01: self-hosted Postgres in both environments for now; PlanetScale later. Production runs on a self-hosted Postgres 17 in the K3s cluster on the droplet, the same way as staging: the CloudNativePG operator, one single-instance cluster per environment in its own namespace, separate from Bifrost’s Postgres, reached by the Workers through Hyperdrive over a Workers VPC Service (no public hostname). Why: create both environments and the pipeline once, with no database bill during the build. Protections, because it holds the money: production data on the droplet’s own disk (a separate DigitalOcean Volume was considered and declined on 2026-11-01; a resize keeps the disk, a rebuild means restoring from R2); continuous WAL archiving plus a daily base backup to a dedicated R2 bucket (point-in-time recovery, 30 days kept); a restore drill before the first real customer (P9.14). Costs accepted: a single node with no failover, so a droplet outage stops paid requests and the dashboard (requests are refused, never served unbilled); it shares the droplet’s CPU and memory with Bifrost; upgrades and backups are ours to run. Moving to PlanetScale later: the app reaches Postgres only through Hyperdrive, so the move is a data copy (logical replication, or dump and restore in a short window) plus a Hyperdrive config change, with no code change; keep to standard Postgres features so nothing blocks it. When is decided at the load test (P9.10): before launch if the cluster Postgres can’t hold the launch load, otherwise when traffic or availability needs it. The move would follow the earlier plan: PS-5 single-node (~$5/month), then HA (from $15/month) before relying on it. Tests are unchanged: a throwaway Postgres container in CI. (Original 2026-10-26: PlanetScale for production from day one.)
Analytics/dashboard read store Start simple: the existing PlanetScale ledger cluster, in a separate schema, not a new instance and not ClickHouse (revised 2026-09-20). Written by the queue consumer in apps/dashboard-api; never the source for billing or rate limiting. Sized for a launch assumption of about 20 req/s or less, likely much lower, not the 100–500 req/s ceiling. Request-log metadata is retained indefinitely. No partitioning at launch, but the schema is built so partitioning, moving to its own instance, or moving to ClickHouse are cheap later. Rules that keep that true: (1) every log row has a non-null created_at, and every query is time-bounded; (2) primary and unique keys on log tables include created_at (e.g. (created_at, id)), because Postgres requires the partition key in any unique constraint, and an id-only primary key would force a table rewrite to partition later; (3) no foreign keys from log tables to ledger tables, ids are plain columns, which also lets the schema move to another instance; (4) log tables are append-only; (5) dashboard aggregates read from rollup tables (e.g. usage by model by hour), not by scanning raw rows; (6) the queue consumer writes through a small store interface; (7) analytics writes use their own Hyperdrive config or connection limit so they can’t starve hold/settle, and dashboard reads use a replica or the rollup tables. The request-logs view reads rows directly, with a pointer to R2 for stored content. Triggers to revisit (indicative, tune later): the largest log table passes roughly 100M rows or 50 GB, dashboard query latency degrades, or analytics load shows up as ledger contention. First step then is partitioning; next is a separate instance; ClickHouse if the aggregate queries outgrow Postgres.
API key security model Keys stored as hashes (never raw), per-key scoping/spend caps to bound blast radius on a leak, explicit rotation/revocation, HTTPS-only, and explicit redaction from request logs given the logging/analytics pipeline already in the design.
Dashboard auth model Better Auth (open-source, MIT-licensed, self-hosted TypeScript auth library), chosen over fully custom and over WorkOS (managed). Social login is a stated requirement (fast signup for developers and target customers), which ruled out fully-custom-from-scratch as the default (est. 1.5–2.5 weeks to build OAuth flows, session rotation, and security hardening per provider). Better Auth provides social login (40+ providers), password auth, and session/refresh-token management out of the box as a library, not a hosted service — estimated integration effort ~2–4 days. Chosen over WorkOS specifically because it keeps user data in Router.Africa’s own PlanetScale Postgres from day one, eliminating vendor migration risk (forced password resets, OAuth app re-registration, no confirmed path to export password hashes if ever moving off a managed provider). Cloudflare/stack compatibility confirmed: runs on Node natively; on Workers, requires Hono as the routing layer (Better Auth’s official Workers support path) — deployed in the dedicated apps/dashboard-api Worker, not the external-facing apps/api Worker (see Worker split decision below). A community integration (better-auth-cloudflare) wires up Workers + Hyperdrive + D1/KV/R2, directly compatible with the existing PlanetScale-via-Hyperdrive setup. Trade-offs accepted: no managed infrastructure — Router.Africa owns uptime, security patches, and library upgrades; API stability across minor versions is still evolving; no SOC 2 certification to hand an enterprise customer’s security questionnaire if that becomes a procurement requirement later. Hard constraint carried from the Niobi web rewrite’s cookie/token bug: whichever credential mechanism is issued must be the only valid auth path end to end in the app’s own request handling — no code path where a stale or leftover credential from a prior mechanism can silently win and authenticate as the wrong user.
Login methods and Better Auth plugins Google, GitHub and GitLab social login, plus email and password. Account linking on: a second provider with the same verified email joins the existing user; only Google is a trusted provider, GitHub and GitLab must report the email verified. Better Auth plugins used: organization (orgs, members, invitations; teams off), admin (staff roles support and admin, ban, revoke sessions, impersonation, see Impersonation), one-time-token, two-factor (optional for customers, required for staff before any admin route works), passkey, haveIBeenPwned (breached-password check), captcha (Cloudflare Turnstile on signup, login and reset) and lastLoginMethod. Deliberately not used: bearer, JWT and API-key plugins (each is a second credential path, against the hard constraint above; API keys are our own), teams, SSO/SAML and the OIDC provider (deferred), anonymous, phone, magic-link and email-OTP login. Tables and ids: db-schema.md section 4.0. (Decided 2026-09-21.)
Dashboard session mechanism (cookies, not JWT) Cookie session, database-backed. Better Auth issues an opaque random session token in an HttpOnly, Secure, SameSite cookie; apps/dashboard-api looks the token up in core.sessions on each request. No JWT is issued to the browser, and the bearer, JWT and API-key plugins are off. Why: (1) the hard constraint in Dashboard auth model: one credential mechanism end to end, so no stale or leftover credential can win (the Niobi cookie/token bug); (2) a database session is revocable instantly, so ban, force logout, log out everywhere and ending an impersonation all work by deleting a row, whereas a JWT stays valid until it expires; (3) an HttpOnly cookie can’t be read by page scripts, unlike a JWT kept in browser storage; (4) each origin already gets its own cookie (app. and admin.), matching Domain structure. Costs accepted: one indexed lookup per request; cookie sessions need CSRF protection, covered by SameSite, Better Auth’s trusted-origin check and CORS scoped to the one origin. Cookie cache (signed session data in a cookie) is off; enable it only if session lookups show up in latency, sessions stays authoritative. No forced logout on a change of location, browser or device (decided 2026-09-21: travellers must not be signed out; the new-sign-in email, session list and step-up cover it), and no credential in web storage. Hardening (__Host- host-only cookies, per-audience secrets and sessions.audience, step-up for sensitive actions, CSRF, no-store, CSP, text-only rendering of stored content, a 20-item conformance suite) is in auth-security.md. (Recorded 2026-09-21. The single-mechanism constraint dates from the auth decision; reasons 2 to 4 were not written down before.)
Public API authentication (OpenAI-compatible) api.router.africa accepts Authorization: Bearer <key> only (scheme case-insensitive; no Basic, x-api-key, api-key, or key in the query string or body; exception, 2026-11-07: POST /v1/messages also accepts x-api-key, see below), so an OpenAI SDK needs only a new base_url and key. Keys are sk-ra- plus 37 random base62 characters and a 6-character CRC32 checksum, shown once, stored as a SHA-256 hash (supersedes the backlog’s ra_live_ idea). The checksum rejects malformed keys before any database lookup and returns the same 401 as any bad key; no customer or workspace id is embedded, since the owner of a leaked key is found by hash lookup. OpenAI-Organization, OpenAI-Project and OpenAI-Beta are accepted and ignored. [Superseded 2026-10-26 by api-errors.md: the error body is OpenRouter’s envelope with metadata.reason, not OpenAI’s shape; and out-of-credit and spend-limit are 402, not 429 (see API error contract and response extras). On POST /v1/messages the body is Anthropic’s format, see 2026-11-07 below.] Error body is OpenAI’s {error: {message, type, param, code}}: no key or wrong scheme is 401; unknown, revoked, expired or pending keys all return one identical 401 invalid_api_key with the key masked; a suspended account is 403; a model that is off or not allowed is one 404 model_not_found. Every response carries x-request-id. Out-of-credit and spend-limit follow OpenAI’s 429 with distinct codes, translated from Bifrost’s 402 at the edge (codes are the error-catalogue decision). apps/api never reads a cookie, never forwards the client’s Authorization to Bifrost, and has no CORS. Full spec and tests: auth-security.md sections 1, 2 and 6. (Decided 2026-09-21; type and code values to confirm against OpenAI’s live responses.)
Impersonation (admin “view as customer”) In v1, reversing its earlier deferral. Staff admin only, customers only, read-only (non-GET routes and stored-content reads are rejected while sessions.impersonated_by is set), 30 minutes, reason required, “Viewing as” banner, impersonation.started and .ended audited, and every audit event in the session carries impersonated_by. The admin and dashboard origins keep separate cookies, so the session is handed across with a single-use token (one-time-token plugin), not a shared cookie. Guardrail details: db-schema.md section 4.0. (Decided 2026-09-21; guardrails proposed, see db-schema.md 8.2.)
Transactional email: Resend All transactional email goes through Resend from apps/dashboard-api (its HTTP API works on Workers): email verification, password reset, invitations, low-balance alerts and receipts. Better Auth sends its own emails through callbacks that call Resend. The sending domain is verified on router.africa (SPF, DKIM, DMARC in Cloudflare DNS); the API key is a Worker secret. (Decided 2026-09-21.)
apps/api Worker routing framework Hono, routing the entire apps/api Worker — ingest, hold/settle, and the Bifrost call. Fills a gap that was never previously decided, not a replacement of an earlier choice — prior discussion of the Worker described it generically without naming a routing framework.
Domain structure Four separate origins, split by consumer and security profile: router.africa (marketing site, Astro), app.router.africa (dashboard SPA + its own backend, including Better Auth — CORS scoped tightly to this one origin), api.router.africa (external, OpenAI-compatible developer API — API-key auth, no CORS since it’s server-to-server only), admin.router.africa (staff-only admin console for rates, markup rules, FX, keys, rate limits, ledger adjustments and audit — its own session cookie, CORS scoped to this one origin, staff flag separate from org roles; added 2026-09-20, see the Build Inventory). Chosen over path-based namespacing (e.g. router.africa/api) specifically to keep the external API’s security boundary clean — a shared /api/* namespace risks a permissive dashboard CORS policy accidentally applying to the money-moving external endpoint. OpenRouter itself uses a path-based pattern (openrouter.ai/api/v1), but this doesn’t constrain compatibility since OpenAI-compatible clients always set base_url explicitly.
Worker split: external API vs. dashboard backend Two separate Workers, not one. apps/api handles only the external developer-facing API (ingest, hold/settle, Bifrost call) at api.router.africa. A new apps/dashboard-api Worker handles Better Auth and the dashboard’s own backend needs (usage stats, billing, key management) at app.router.africa. Chosen over keeping both in one Worker (which was the prior default) for isolation: the external hot path and the auth/session logic no longer share deploys or blast radius, and each Worker’s middleware (CORS, auth type) is scoped to what it actually needs rather than requiring route-group-level care to avoid cross-contamination.
Queue consumer placement Folded into apps/dashboard-api as a queue() handler, not a standalone apps/queue-consumer Worker. Cloudflare Workers support both an HTTP fetch() handler and a queue() handler in the same script, so a separate deployable wasn’t required to keep the async-decoupling benefit. apps/dashboard-api is the natural owner since it’s already the Worker reading usage/billing data back out for the dashboard — the same Worker now writes it. Trade-off accepted: this couples the queue consumer’s deploy to the dashboard-api Worker’s deploy; if analytics-processing volume ever needs to scale or deploy independently of the dashboard’s own backend traffic, splitting it back into its own Worker is the natural next step — not a starting assumption at current scale (100–500 req/s).
Scheduled jobs Scheduled work runs as Cloudflare Cron Triggers on apps/dashboard-api, alongside its queue() handler, not in a separate Worker: the daily rate computation (Bifrost cost, FX, markup, round up, publish, only for models that are on), the model catalog sync from Bifrost, key reconciliation between Postgres and Bifrost (feeds the admin discrepancy view), and low-balance alerts. Same trade-off as the queue consumer: deploys are coupled, and splitting out is the natural next step if volume demands it. (Decided 2026-09-20.) Refined 2026-09-20: the jobs are not each their own Cron Trigger. See Scheduled jobs admin page.
Data residency / KYC / AML Out of scope for Router.Africa’s own compliance work — handled via Niobi’s existing API signature/auth mechanism.
Repo structure Single Turborepo monorepo: apps/api (external API Worker, api.router.africa), apps/dashboard-api (Better Auth + dashboard and admin backend + queue consumer, app.router.africa and admin.router.africa), apps/dashboard (Vite frontend, served from app.router.africa alongside its backend), apps/admin (separate Vite frontend for admin.router.africa, its own build and deploy so admin code never ships in the customer bundle; talks to the shared apps/dashboard-api backend), shared packages/types, packages/db, packages/config. Bifrost’s config (not source) lives under infra/gateways/<instance>/ (revised 2026-10-25, Gateway abstraction); Bifrost itself is not part of this repo.
Testing approach Vitest for units (matches existing Vite tooling). @cloudflare/vitest-pool-workers for integration tests — required specifically because Workers run in a V8 isolate with different constraints than plain Node, so plain-Node integration tests would give false confidence. Real Postgres for DB integration tests rather than mocking it, since the hold/settle logic’s correctness depends on real row-locking behavior: a throwaway Postgres container in CI, same major version as production (revised 2026-10-26; originally PlanetScale branching). k6 for load testing. Playwright for end-to-end tests, run by GitHub Actions against staging (added 2026-10-26). A dedicated concurrency-correctness test (N concurrent requests against one account, assert no over-approval) is treated as the single highest-priority test in the suite.
Threading/concurrency model No manual thread/pool sizing needed anywhere in this stack — Workers and Node use an event-loop model (async I/O, not thread-per-request), Cloudflare auto-scales isolates, Bifrost uses goroutines internally. Hyperdrive’s connection pool size is the one real capacity number to watch.
Billing/pricing model See the dedicated Pricing & Markup Model document (backlog item #11): daily published per-model, per-token-type rate in local currency, from Bifrost’s highest-cost-across-providers estimate, FX from the providers in FX providers and rate basis, a scoped flat/percentage markup rules engine, per-workspace percentage discounts applied at charge time, and round-up. Rate locked at hold time, used for settle. No direct provider integration beyond Bifrost. The actual launch markup values are set later in the admin app (see Markup configuration), not a build blocker.
Idempotency key design Two-part design, since OpenAI’s own chat/completions spec has no idempotency concept to build on (client SDKs won’t send one by default). Part 1 — automatic, no client action required: on every request, before the hold write, compute hash(account_id + exact request body) and attempt INSERT ... ON CONFLICT DO NOTHING into an idempotency_keys table (hash, account_id, status, response_body, created_at) within the same transaction as the hold, not a separate write — this keeps the cost at effectively zero given PlanetScale’s flat cluster pricing, versus the ~$1,300–6,500/month a Cloudflare KV-based version would have cost at stated volume. Dedup window is short (seconds, not 24h) — long enough to catch a genuine SDK-level retry after a network blip, short enough not to falsely block a legitimate identical prompt sent minutes later. If the insert conflicts: an in_progress original → return 409 Conflict; a completed original → return its stored response, skip hold/settle entirely. Part 2 — optional, client-supplied: an Idempotency-Key header, documented explicitly as a Router.Africa extension (not part of OpenAI’s spec) in the developer docs (#10), settable via extra_headers in any OpenAI SDK, for developers who want an explicit, longer-lived guarantee beyond the automatic short-window protection. Not a substitute: Bifrost’s semantic/exact-hash caching is a different mechanism (reduces Bifrost’s own compute cost on similar/identical prompts) and does not protect Router.Africa’s ledger — without the dedup above, a client retry hitting Bifrost’s cache would still produce a second hold/settle against the customer even though Bifrost only did the work once.
Mid-stream client disconnect policy Checked Bifrost’s own behavior before designing this (maximhq/bifrost release notes) rather than building from scratch. Bifrost already has proactive SSE disconnect detection (v1.6.8 — detects disconnects actively during streaming, not only on a failed write) and is actively engineering correct partial billing on cancelled streams (“Billing on Failed Streams” fix, confirmed for Anthropic and Bedrock specifically — not confirmed complete across all 15+ providers Bifrost fronts). Design: trust Bifrost’s reported partial usage (usage.total_tokens, usage.cost) as the primary source for settling a disconnected request — not Router.Africa independently counting streamed tokens as the ground truth. Router.Africa’s own token count is kept as a reconciliation/sanity check, not the primary mechanism, specifically because provider coverage for Bifrost’s partial-billing fix isn’t confirmed complete — a divergence between Bifrost’s reported number and Router.Africa’s own count on a given provider is a signal to investigate, not something to silently trust either source blindly for.
Worker → Bifrost timeout/retry policy Checked Bifrost’s own behavior before designing this. Bifrost already implements bounded per-provider timeout (not accumulating per-retry the way naive proxies do — e.g. ~2s max per provider attempt, so a two-provider fallback chain completes in ~4s, not 30-60s) and its own configurable retry/fallback across providers (max_retries, retry conditions, explicit fallback chain) — confirmed explicitly that a provider failure only becomes a request failure if no fallback is configured. Design, revised from an earlier version that duplicated this at the Worker level: the actual decision is how to configure Bifrost’s own timeout/retry/fallback settings (per-provider timeout, retry conditions, fallback chain), not a separate Worker-level retry on top of Bifrost’s internal one — a Worker-level retry stacked on Bifrost’s own retry/fallback risks compounding rather than helping. The Worker’s role simplifies to: set an overall request timeout comfortably longer than Bifrost’s worst-case internal fallback time, and handle whatever Bifrost ultimately returns — success, or a final failure after Bifrost has already exhausted its own configured resilience (release the hold on final failure, no further Worker-initiated retry).
Rate limiting / abuse prevention Enforced in Bifrost, configured from Postgres. Every Router.Africa API key maps 1:1 to a Bifrost virtual key (see API key ↔ Bifrost virtual key mapping), so Bifrost’s own per-VK request/token rate limits do the enforcement. Bifrost doesn’t know about Router.Africa users, so Postgres holds the limit configuration per organization (decided 2026-09-20, replacing per-key; set in the admin app) and apps/dashboard-api pushes it to Bifrost’s governance API. Bifrost enforces rate limits at the virtual-key level only (no aggregate across keys), so the org’s limit is copied onto each of the org’s keys’ virtual keys: new keys inherit it at creation, and changing an org’s limit fans out to all its keys through the same sync path (drift shows in the admin discrepancy view). Accepted: the limit is effectively per key, so an org with many keys can exceed it in aggregate; revisit if that is abused. Limits remain admin-only, invisible to the customer (see Dashboard Scope). Supersedes the earlier design — a Postgres counter table (account_id, window_start, count) checked in the same transaction as the hold and idempotency claim — which is dropped: the hold transaction now covers only the idempotency claim and the hold. Consequence: a Bifrost 429 arrives after the hold is written, so the Worker must release the hold on any Bifrost rejection (same path as a final provider failure). Rate limiting still must not be read from the async request/analytics log — its write lag means a burst isn’t visible until it has already gone through.
Request logging Every API request logs metadata — timestamp, account, endpoint, model, status, latency, tokens, cost — feeding the dashboard’s request-logs view and the audit trail. This is the existing async pipeline (fire-and-forget to Cloudflare Queues, written by apps/dashboard-api’s queue consumer) — already decided under Async/Post-Response Work, restated here explicitly as a schema requirement (#6) rather than left implicit. Explicitly not the source for rate limiting — see the entry above for why the async write lag makes it unsuitable for a real-time check.
Observability tooling Datadog, chosen over Sentry — a stronger fit than initially assessed, not just preference: Bifrost has a native Datadog plugin (APM traces via dd-trace-go, already built, not a bolted-on SDK), Cloudflare Workers has native OpenTelemetry export (automatic instrumentation, exports to any OTel-compatible backend — Datadog included), and apps/niobi-proxy (TypeScript on Node, per the Niobi payments decision) can use Datadog’s Node APM library (dd-trace) or OTel export. This means Datadog serves as one genuinely unified platform across the whole stack — Bifrost via its native plugin, Workers via OTel export, niobi-proxy via the Node APM library — which removes the earlier “defer a unified view until later” caveat entirely; Datadog gets you that unification natively, not as a future project. Cost tradeoff worth naming: Datadog is typically pricier than Sentry at scale — acceptable given the architecture fit and stated preference, but worth watching as usage grows.
CI/CD pipeline Revised 2026-10-26: one pipeline, GitHub Actions only. Replaces the earlier two-pipeline design (Cloudflare Workers Builds and Pages Git integration for Cloudflare targets, GitHub Actions for K3s). Why it changed: two systems reacting to the same push independently can’t guarantee migrate first, then code or that apps/dashboard ships with apps/dashboard-api, and Workers Builds has no notion of deploying a chosen release tag to production on demand (Release and branch flow). How it works: GitHub Actions runs every deploy in a fixed order: tests and gates → database migration → apps/dashboard-api → apps/dashboard (Pages, via wrangler pages deploy) → apps/admin (Pages) → apps/api → droplet services (apps/niobi-proxy, gateway config) through the self-hosted runner. Cloudflare only receives wrangler commands; Workers Builds and the Pages Git integration are not connected. A merge to main runs it against staging automatically; a manual Run workflow on the production workflow, given a release tag, runs it against production. Each Worker still has -staging and -production environments with their own [env.*] Wrangler blocks, bindings and secrets; K3s targets use staging and production namespaces. PR previews of the dashboard come from wrangler pages deploy --branch. Cost: stays on GitHub Free (2,000 hosted-runner minutes a month for a private repo; self-hosted runner minutes are free). Not adopted: Jenkins (another server to host, patch and secure, and it duplicates Actions; reconsider only for heavy multi-machine test orchestration). End-to-end (Playwright) and load (k6) tests run on Actions too; long suites move to a self-hosted runner off the production droplet if minutes run short. (Original decision superseded; see cicd.md.)
Self-hosted runner and droplet deploys (2026-11-05) Operator decisions after GitHub Free’s hosted minutes ran out (2026-11-04) and stopped every job. (1) No paid GitHub plan, and no Jenkins. Jobs on a self-hosted runner use no hosted minutes, so every job of a deploy runs on the runner in the cluster (P0.09), the gates included. PR checks stay on GitHub-hosted runners within the free minutes; when those run out, PR checks wait for the next month and deploys carry on. Jenkins (already set aside under CI/CD pipeline) would need the same machine plus its own server and a rewrite of every workflow. (2) One runner pod, not actions-runner-controller: lighter on the 4 GB droplet; one job at a time. (3) The runner may change named objects only, get and patch, in default (route-split), cloudflared and postgres-production (the db-backup CronJob only), with no access to any Secret. So the pipeline applies those charts with helm template and kubectl, not helm upgrade, which needs every Secret in its namespace. (4) Because one cluster serves both environments, these shared releases change with a production deploy only; a staging deploy shows the diff. Accepted: anyone who can push to main can change what the runner runs (GitHub Free has no branch protection; Release and branch flow). Detail: cicd.md section 1, Reaching the cluster.
Niobi payments static IP handling Niobi’s payments integration requires server-to-server calls from a static IP for allowlisting. Cloudflare Workers have no static outbound IP by default (egress can originate from any Cloudflare edge IP). Resolved with a small, separate relay service (apps/niobi-proxy, TypeScript) on the existing K3s droplet — apps/dashboard-api calls it over a private connection (now decided: Workers VPC binding over a Cloudflare Tunnel, see Private service access), and the relay makes the actual outbound call to Niobi using the droplet’s stable IP. Deliberately minimal scope: a relay only — verify the caller, forward to Niobi, return the response; no business logic (that stays in apps/dashboard-api). Language: TypeScript, not Go — initially considered for consistency with Bifrost on the same droplet, but reconsidered since the droplet doesn’t require every service on it to share a language; TypeScript instead enables sharing request/response types with apps/dashboard-api via packages/types. Security requirements: caller verification (now decided: HMAC signature, see Private service access) and Niobi’s own credentials held here, separate from apps/dashboard-api’s secrets. Staging support: must be able to mock Niobi calls (env-flag-driven mock mode within the same service) so staging doesn’t need live payment credentials or risk real transactions. Signing implementation: Niobi’s official reference implementation/SDK for request signing exists and will be ported into apps/niobi-proxy (exact integration details to follow) rather than implemented from a spec from scratch. Containment requirement: all Niobi signing logic — the algorithm, the credentials it uses, everything related to authenticating to and from Niobi — lives exclusively inside apps/niobi-proxy; apps/dashboard-api and every other service in the stack never sees or handles it directly, only calls the relay. Inbound verification placement: verifying signatures on inbound Niobi-authenticated traffic also happens inside apps/niobi-proxy, not as a gatekeeping step in front of the Worker or inside another service’s auth flow — consistent with keeping all Niobi-specific auth logic (both directions) in one place. Chosen over Cloudflare’s native fix (Workers VPC binding + Gateway egress with a dedicated IP) because that requires Enterprise-tier Cloudflare, premature spend for one integration point; also chosen over a third-party static-egress-IP proxy service (e.g. QuotaGuard) to avoid a fourth vendor relationship for something the existing droplet already solves.
Model availability enforcement Raised by the Dashboard Scope doc’s onboarding flow (customers choose which models are enabled for their account/key at signup, editable later). This is a backend enforcement requirement, not just a frontend selection — apps/api must check the requested model against the account’s/key’s allowed_models list before routing to Bifrost, rejecting or falling back appropriately if a request targets a model outside the selection. Sits alongside the idempotency/rate-limit/hold transaction already decided in the request path. Scoped per key — allowed_models is a field on the API key record (see Model selection scope). Because each key maps 1:1 to a Bifrost virtual key, the same list can also be pushed to Bifrost’s deny-by-default model allowlist as a second line of defence.
API key ↔ Bifrost virtual key mapping Router.Africa issues its own API keys (hashed in Postgres, per the API key security model) and maps each 1:1 to a Bifrost virtual key, so Bifrost’s governance features — rate limits, budgets, model allowlists — apply per key without being rebuilt. The Worker authenticates the customer’s key against Postgres, then calls Bifrost using the mapped virtual key. apps/dashboard-api provisions, updates and revokes the virtual key alongside the Router.Africa key. Resolves the design doc’s “per-customer virtual keys” wording against the own-keys model above: both are true, linked one-to-one. Follow-ups this creates: the mapping needs a defined sync/failure path (key created in Postgres but not in Bifrost, and vice versa), and it reopens the currency-unit question in openrouter-api-comparison.md #30 — Bifrost budgets are denominated in USD while the ledger is local currency, so decide which system is authoritative for spend caps.
Key ↔ Bifrost sync and failure path A key is never shown to the customer until its Bifrost virtual key exists. Postgres and Bifrost can’t share a transaction, so “atomic” is implemented as an ordered create with compensation, in apps/dashboard-api: (1) insert the key row in Postgres as pending (hash only, not yet usable for auth); (2) create the Bifrost virtual key with the key’s rate limits and allowed_models; (3) on success, mark the row active and only then return the raw key to the frontend; (4) on any failure, delete or roll back whatever was created on either side and return an error, with no key shown. A pending key can never authenticate, so a crash between steps leaves no usable half-key. Updates (rate limit, models) follow the same order and revert on failure. Revocation marks the key revoked in Postgres first (auth is Postgres-side, so the key is dead immediately), then deletes the Bifrost key with retry. Admin discrepancy view: a section in the admin app lists every mismatch: keys stuck pending, keys active in Postgres with no Bifrost key, Bifrost keys with no Postgres row, revoked keys still live in Bifrost, and config drift (rate limits, allowed models, admin cap differing between the two). Each row supports re-sync or cleanup by an admin, and actions are audit-logged. A periodic reconciliation job populates the view. (Decided 2026-09-20.)
Spend cap authority Two caps, two owners, split by who sets them. The admin-set cap (set by staff in the admin app) is owned and enforced by Bifrost as a virtual key budget, denominated in USD, and pushed by apps/dashboard-api with the key. The user-set cap (set by the customer for their account or key) is owned, calculated and enforced by Postgres in credits, against the ledger, in the hold path. Neither system tries to enforce the other’s cap, so no USD/local-currency conversion of a customer-visible amount is needed. A request must pass both. Resolves openrouter-api-comparison.md #30. Consequence: the admin cap’s USD figure is set by staff, not derived from the ledger, so it won’t track FX moves on its own. Staff set it as a backstop limit, not a precise customer entitlement. (Decided 2026-09-20.)
v1 model list Not a fixed list; it’s configuration. Models are configured in Bifrost. The admin app reads the configured models from Bifrost’s model listing endpoint (to be verified against Bifrost’s API) and shows them. Staff choose the default models available to all users there, which also sets the default pre-checked models in onboarding. Per-key allowed_models (see Model selection scope) stays the enforcement field, so staff add or remove models for a specific customer or key through the admin app, which writes Postgres and pushes to Bifrost per the sync path above. A model is “on” only when staff have explicitly enabled it and it has a published rate (Pricing & Markup Model). Enabling is the single gate: staff set the rate first, then turn the model on. Customers can only add or select models that are on, and apps/api rejects requests for any other model before reaching Bifrost, so usage on a model with no pricing (Bifrost reports cost 0.0 when it has no pricing entry) can’t happen. Bifrost provider keys use explicit model lists, never ["*"], so Bifrost also denies anything not turned on. Newly released models appearing in Bifrost’s catalog via its 24-hour pricing/model sync do not become available until staff turn them on; the default scheduled sync is accepted. Bifrost upgrades are for features, not for new models. The Maxim datasheet URL is a hosted dependency; Bifrost now makes BIFROST_MODEL_PARAMETERS_URL configurable (UI, config.json, Helm), so an override is possible if egress is ever restricted. Closes the split-out question from Launch scope. (Decided 2026-09-20.)
Ledger source of truth Router.Africa’s own Postgres ledger holds money: balances, holds, settlements. Bifrost’s governance layer (virtual keys, budgets, rate limits) is used for enforcement on the gateway side, not as the ledger. This closes the evaluation flagged in infra/plugins-and-extensibility.md and the README (“how much of Bifrost’s governance can be reused before building custom ledger logic”) — answer: reuse it for rate limits, budgets and allowlists via the 1:1 key mapping; build the currency ledger and hold/settle ourselves.
Price source of truth / repricing cadence The design doc (§5) governs balance semantics: balances are held as credits (equivalent to tokens, never labelled with a currency, non-refundable), priced at time of consumption, with repricing applying immediately and uniformly (no grandfathering). The Pricing & Markup Model doc governs the mechanics: a daily published rate, locked at hold time. The business model doc’s monthly review cadence is superseded by the daily rate. The business model’s price table and margin figures are illustrative history, not the current price sheet.
Audit trail & prompt/response retention An audit trail is a v1 requirement, correcting the design doc’s earlier non-goal (“no audit logs”). Request metadata is logged as decided under Request Logging. Prompt and response content is NOT saved by default; saving is opt-in (revised 2026-11-07, superseding the 2026-09-20 “saved by default, opt-out” decision and returning to the earlier opt-in position; this matches the published Terms 8.4, the Privacy policy 3.3/8 and the website copy). The setting is per workspace (surfaced under Data & privacy for the current workspace, see Dashboard Scope), off when a workspace is created, and changes are audit-logged with a timestamp. Turning it off stops future capture. Retention is indefinite for now; a retention window, customer deletion of stored content, and whether opting out purges existing content are deferred to future-optimizations.md (the key layout supports all three). Redaction: no PII redaction in v1 (unreliable, easy to oversell). Message bodies only are stored; headers and API keys are never stored, and content is kept out of Datadog traces and Bifrost’s own logging. Access: stored content is readable only by admins, both Router.Africa staff admins and customer org admins; other roles, including members, have no access to stored content. Every read of stored content is audit-logged. Audit trail store: not a separate store; audit events live in the same store as the request log (decided). Because capture is opt-in, the dashboard toggle, terms and privacy policy must describe it plainly and the consent timestamp and terms version are recorded when it is turned on. Storage: Cloudflare R2, keyed by request ID with a pointer on the request log row, written by the Worker after the response (not via the usage queue, whose message size is too small for content), only for workspaces that have opted in. Write path (start simple): one object per request holding prompt and response together, gzip-compressed, written once after the response completes and off the hot path, behind a small write interface. Key layout {account_id}/{yyyy}/{mm}/{dd}/{request_id}.json.gz, so per-customer deletion or export is a prefix operation. Streaming responses are buffered and written when the stream ends; on client disconnect, write what was buffered and mark the row partial. A maximum stored size per request is set, with anything larger truncated and flagged. Workspaces that have not opted in write nothing. The pointer on the request-log row is an object key plus optional offset and length columns (empty at launch), so batching many requests into one object later needs no schema change; revisit when R2 write costs pass roughly $100 a month (about 10 req/s sustained at $4.50 per million writes). Infrequent Access storage is not used at launch (writes cost double, 30-day minimum). Bytes and object counts are tracked per account in the rollup tables. Token counts and cost stay in Postgres, never in R2. A failed content write must never fail the request. Message bodies only; headers and keys are never stored. Bifrost’s own content logging (disable_content_logging) stays on the disabled setting: it is a global switch (not per key), has no documented retention or delete path, and would write every customer’s prompts to the droplet Postgres regardless of the workspace setting. Revisit if Bifrost adds per-key logging and deletion. (Decided 2026-09-20; default revised 2026-11-07: not saved by default, opt-in.)
Launch scope (markets) All markets live with Niobi at launch — no phased rollout by market. The v1 model list is a separate question and remains open.
Model selection scope Per key. A customer’s model selection lives on each API key (allowed_models on the key record), so onboarding’s model-selection step applies to the first generated key, and the API Keys screen edits it per key. Closes the previously open account-vs-key question.
Dashboard and dashboard-api deploy Together — apps/dashboard and apps/dashboard-api deploy as one unit from the monorepo. Closes the open sub-decision in the tech stack. Enforced by the single pipeline, which deploys dashboard-api then the dashboard in the same run (CI/CD pipeline, revised 2026-10-26).
Marketing site location Already built, in a separate repo (Astro), not part of this monorepo. The copy in docs/product/website-copy.md is source material for it.
Charge basis vs cost basis The customer is charged on the token counts Bifrost reports (input and output separately) × the published rate locked at hold time. Bifrost’s usage.cost is recorded per request as Router.Africa’s actual cost (COGS), for margin tracking and reconciliation only; it never determines what the customer is charged. Router.Africa does not count tokens itself for billing.
Credit balance & non-refundability Balances are presented as credits, equivalent to tokens, and are never displayed with a currency symbol or code (dashboard, API, docs, copy). This avoids implying unspent balance is refundable cash. Credits are consumed by usage only. Ledger amount at top-up is 1:1 with the credit amount purchased.
Top-up transaction fee (Niobi markup) Customers choose a credit amount; the collection request is for credit amount + transaction fee (e.g. 1,000 credits + 40 fee → 1,040 collected). Only the credit amount is credited to the balance; the fee is recorded separately as revenue and excluded from the customer-balance side of reconciliation. Fee values are configured per vendor in the admin app, not hardcoded. The fee breakdown is shown before the customer confirms. Distinct from the per-token model markup below.
Markup configuration The system is built to configure markup every supported way (flat and percentage; global, currency/market, model scopes — per the Pricing & Markup Model; account-scoped and tiered markup were dropped 2026-09-20, see Workspace discounts). The launch values are not a build dependency: the schema and rules engine ship first, and the actual amounts and percentages are entered later through the admin app as configuration (decided 2026-09-20). They must be set before the public price sheet is published.
Workspace discounts Rates, FX and markup are never per workspace. Workspace-specific pricing (negotiated or volume) is a percentage discount on usage, set per workspace, for all models or one model (model-specific wins, no stacking), on or off with a reason, audit-logged. Charge = Bifrost-reported tokens × published rate × (1 − discount). The discount is locked with the rate at hold (at request start when hold is off), and the hold is priced the same way. Replaces account-scoped and tiered markup in the Pricing & Markup Model; a volume tier can return later as a tiered discount. (Decided 2026-09-20.)
Hold close-out Settle and release each close their hold in the same transaction as the balance change, so a completed request never leaves a hold behind. No hold expiry and no sweeper job. A Worker that dies between hold and settle leaves an open hold and a raised held; the ledger invariant check flags the stuck hold, and the usage reconciliation (next entries) finds the uncharged call, which an admin charges or releases. (Decided 2026-09-20.) Built 2026-11-04 (code #96, #99, #102): settle and release close the hold first and move the balance only if they closed it; releasing an unbilled row also releases its hold; a hold the gateway never logged is released from the admin Stuck holds list once it is over 60 minutes old (STUCK_HOLD_MINUTES), admin role and reason required, audit event hold.released, refused while an unbilled row for it is open. Every close-out also gives back the key’s spend-window reservation (holds.window_start, window_reserved). Still no expiry and no sweeper.
Top-up status check Pending top-ups are resolved by a scheduled workflow that queries the transaction status from Niobi through niobi-proxy, alongside the callback. Both paths run the same guarded code (UPDATE ... WHERE status = 'pending'), so a top-up is credited once. A top-up still pending past a threshold is flagged in the admin Transactions view; nothing auto-fails. (Decided 2026-09-20.)
Log-write and audit rules The queue consumer is idempotent: INSERT ... ON CONFLICT (created_at, id) DO NOTHING, rollups incremented only when the insert happened, created_at taken from the event. Privileged audit events (ledger adjustments, key and hold-toggle changes, content-logging and discount changes, releasing a stuck hold) are written directly by apps/dashboard-api, not through the queue. A stored-content read writes its audit event before returning content; no event, no content. Content deletion, when it ships, is recorded as a content.deleted audit event and never updates the request-log row. (Decided 2026-09-20.)
Ledger invariant check A scheduled check per workspace: balance = SUM(ledger_entries.amount) and held = SUM(open holds); any difference alerts. Expired idempotency rows, sessions and verification tokens are deleted by the same housekeeping run. (Decided 2026-09-20.) Since 2026-11-04: the check also proves usage_charges_daily equals the ledger’s settlement entries per workspace (code #98). The purge deletes in batches of 5,000 and counts in Postgres, so a large backlog of idempotency rows (each holds a compressed response) never reaches the Worker or holds locks for long (code #104).
Usage reconciliation (Bifrost vs ledger) An hourly Cron Trigger on apps/dashboard-api finds calls Bifrost served that the ledger never charged. apps/api sends our request id to Bifrost as x-request-id on every call, so a Bifrost log’s id is our request id. The job pages Bifrost’s GET /api/logs (start_time, end_time, min_tokens=1, limit up to 1000) from its last watermark up to now minus a 15-minute grace, and checks each log id against core.settlements.request_id. No settlement: a not_charged row in usage_discrepancies. Settlement with different token counts: a token_mismatch row (the sanity check the disconnect policy calls for). An admin resolves a not_charged row by charging it or releasing it without charge, reason required, audit-logged. Charging uses the normal settle code path at the rates and discount locked on the request’s open hold; with no hold (hold off), the published rate in effect at the Bifrost log’s time and the workspace’s current discount. The settlement is flagged source = 'recon' and the hold is closed. Nothing is charged automatically; automatic charging for calls with an open hold is a possible later step. Bifrost still logs tokens, cost and status with disable_content_logging on, so content logging stays off. To verify in staging before relying on it: x-request-id becomes the Bifrost log id, and log rows carry the virtual key id. (Decided 2026-09-20.)
Scheduled jobs admin page The admin app lists every scheduled job with its status and lets staff change when it runs and switch it on or off. Cron Trigger schedules can’t be edited at runtime, so apps/dashboard-api has one Cron Trigger firing every minute (a dispatcher), and each job’s schedule (cron expression, UTC), enabled flag and config live in core.scheduled_jobs. The dispatcher runs whatever is enabled and due, holding a lease per job so runs never overlap (a lease older than the job’s max runtime is treated as a dead run and reclaimed). Each run is recorded in logs.job_runs (started, finished, status, summary counts, error). Screen: per job last run, last status, next run, running now, recent run history; edit schedule and config (validated); enable or disable (reason required, warns that disabling means stale rates, undetected unbilled usage, and so on); Run now. Admin role can edit, Support role read-only. Every change and manual run is audit-logged. The job set is fixed; jobs are not added or removed from the screen. Minimum schedule granularity is one minute. (Decided 2026-09-20.)
Private service access (Bifrost, niobi-proxy) Our Workers reach both droplet services the same way: Cloudflare Workers VPC Service bindings over a Cloudflare Tunnel. cloudflared runs in the K3s cluster and only dials out; neither service gets a public hostname or DNS record for this path, so nothing on the internet can address them. Bindings are scoped per service (one host and port each): apps/api binds Bifrost only; apps/dashboard-api binds Bifrost (key sync, model list, usage reconciliation) and niobi-proxy. Each service still checks the caller: Bifrost requires the virtual key (enforceAuthOnInference: true); niobi-proxy verifies an HMAC-SHA256 signature over method, path, timestamp and body hash, with a 5-minute window (the secret is a Worker secret on dashboard-api and a Kubernetes secret on the droplet). Every call goes through one client module per target, so the transport can change in one file. Bifrost stays invisible to customers: apps/api strips Bifrost-identifying response headers and maps Bifrost errors to our OpenAI-format errors. Status: Workers VPC is in beta, available on Free and Paid plans at no extra cost during the beta; post-beta pricing is not published (checked 2026-10-25). Fallback if it is withdrawn or priced out: a Tunnel hostname behind Cloudflare Access with a service token per Worker. Access plus service tokens was rejected as the primary because it publishes a hostname. This replaces “e.g. Cloudflare Tunnel” and “shared secret or mTLS” in Niobi payments static IP handling. (Decided 2026-10-25.)
Droplet network lockdown Target state: the DigitalOcean Cloud Firewall (outside the VM) blocks all inbound traffic except the WireGuard UDP port and 443 from Niobi’s webhook IPs (see Niobi webhooks). Outbound stays open (tunnel, providers, Niobi). ufw is not relied on, because K3s writes its own iptables rules for Traefik, ServiceLB and NodePorts that can go around it. Operator access (SSH, kubectl, Bifrost dashboard, Bifrost’s Postgres, Rancher later) is over WireGuard only. Chosen over Tailscale: operator preference, no third-party coordination server, and WireGuard does not answer unauthenticated packets. The public prod-bifrost-core.router.africa ingress and its DNS record are removed. The recovery path is the DigitalOcean web console, tested before SSH is closed. Timing: the public hostname stays during the build; lockdown is a pre-launch task, done before real provider keys carry real traffic. Because the Workers use the VPC path from day one, lockdown is infra-only with no code change. (Decided 2026-10-25.)
Niobi webhooks Niobi callbacks go straight to apps/niobi-proxy; no other surface talks to Niobi in either direction. The DigitalOcean firewall allows 443 only from Niobi’s fixed source IPs. The webhook hostname is DNS-only (not proxied by Cloudflare), so the firewall sees Niobi’s real IP. TLS uses cert-manager with a DNS-01 challenge through a Cloudflare API token, since HTTP-01 cannot work with port 80 closed. Traefik routes only the webhook path to niobi-proxy. niobi-proxy verifies the signature (already decided), then pushes the verified event to a Cloudflare Queue (niobi-callbacks) through the Queues HTTP API, using an API token limited to sending queue messages; apps/dashboard-api consumes it and runs the guarded credit path (Top-up status check). dashboard-api therefore has no public callback route. Chosen over an HMAC-signed POST to a dashboard-api route (adds a public internal route) and over dashboard-api polling niobi-proxy (up to a minute’s delay). (Decided 2026-10-25.)
Admin origin access admin.router.africa (frontend and admin API routes) sits behind Cloudflare Access (Zero Trust, free up to 50 users), so only staff identities can load it. Better Auth, the staff flag and required two-factor still apply behind it. Policy (decided 2026-10-26): an allow-list of named people in Cloudflare Zero Trust (at launch, the operator only), login by Cloudflare’s email one-time PIN; strength rests on those email accounts, covered by Cloud account security. (Decided 2026-10-25.) Revised 2026-11-02: the login method is Cloudflare login, not the email one-time PIN (set up by the operator on the staging admin, P8.01); production (P9.04) uses the same. The Access application also covers *.router-africa-admin.pages.dev, so the Pages address can’t bypass it, and preview deployments require Access.
Bifrost virtual key secret storage Closes the gap in api-key-security.md section 5. The virtual key’s value (generally: the gateway credential’s secret) is stored encrypted in api_key_gateway_credentials.secret_enc: AES-256-GCM via WebCrypto, a random 96-bit nonce per value, and the key id plus gateway id as associated data, so a value cannot be moved to another row. The encryption key is a Worker secret held by apps/dashboard-api (encrypts at key creation) and apps/api (decrypts per request). enc_version names the encryption key used, so it can be rotated by re-encrypting rows in the background. The value is never logged, queued or returned by any route. (Decided 2026-10-25.)
Launch shape Public launch with open signup. No closed beta: friendly customers are onboarded at launch, on the same public product. Everything in build-inventory.md (all screens and features) is in launch scope; its Later list stays later. (Decided 2026-10-25.)
Gateway abstraction (Bifrost is swappable) Bifrost is the current gateway, not a fixed part of the design. Why: (1) vendor risk: if Bifrost stops shipping updates or changes terms, we can move without a rewrite; (2) later scale or isolation: a second instance for failover, or a dedicated instance for a large customer. Neither is expected near term (Bifrost’s throughput is well above the 100–500 req/s target; the droplet, PlanetScale and provider limits bind first), so the foundations are built now and the routing behaviour is deferred. Built now: (a) code: apps/api and apps/dashboard-api talk to the gateway only through one Gateway interface (chat completion including streaming, create/update/delete a key credential, list models, fetch usage logs for reconciliation, map errors to our OpenAI-format errors); Bifrost is the only adapter. (b) data: a gateways table (one row at launch, separating instance from adapter type, so bifrost-1 and bifrost-2 can coexist); the key’s gateway credential lives in api_key_gateway_credentials keyed by (key_id, gateway_id); columns holding gateway-reported numbers are named gateway_*, and gateway_id is recorded on request logs, settlements-adjacent reconciliation rows and rollups, the large tables that should never need a migration for this. (c) source of truth: Postgres holds every key, limit, budget and allowlist, and the key sync job can rebuild a fresh gateway from it, so the gateway holds nothing we can’t recreate. (d) infra: gateways live under infra/gateways/<instance>/, each with its own Kubernetes namespace, Helm release, Workers VPC Service and Worker binding (gateways.binding). (e) version pinning: the exact Bifrost image we run is pinned and a copy kept. At launch there is exactly one gateway; every org, key and model uses it, and no code chooses between gateways. Deferred (future-optimizations.md): assigning orgs to instances and moving an org between them, per-instance model lists, failover across two instances, per-model routing across gateway types. What any gateway must support: per-key credentials, rate limits, budgets and model allowlists, usage logs that carry our request id and token counts, and an OpenAI-compatible endpoint, because rate limiting and the admin USD cap stay enforced in the gateway (Rate limiting, Spend cap authority). A plain model server (OpenLLM, vLLM, Ollama) does not qualify; it is added as a provider behind a gateway. Moving those controls into our Workers was rejected for now (it means building and load-testing rate limiting on the hot path and reopening two decisions). In other docs, “Bifrost” means “the gateway (currently Bifrost)”. (Decided 2026-10-25.)
Release and branch flow Trunk-based with release branches and semantic versions (major.minor.patch). main is the only long-lived branch: PRs merge into main, and every merge deploys to staging automatically. Production is always intentional: cut release/<major>.<minor> from main, tag it v<major>.<minor>.0, then run the production workflow for that tag. Hotfix: the fix merges to main first, is cherry-picked into the release branch, tagged v<major>.<minor>.<patch+1> and deployed the same way, so the next release never drops it. Hotfixes avoid migrations; an unavoidable one lands on main first and is cherry-picked unchanged (the database is forward-only). Branch rules are not enforced (revised 2026-11-01): GitHub Free doesn’t enforce branch protection on a private repo, and GitHub Pro ($4/month) was declined. Instead, nobody pushes directly to main or release/* (a rule in the code repo’s CLAUDE.md for agents), every change goes through a PR, and the PR checks still run on every PR; merging with a failing check is a choice the merger has to make deliberately. Revisit when the second developer joins. On GitHub Free with a private repo there is no deployment-approval step; the manual Run workflow is the deliberate step, and only the operator deploys at launch. No staging branch. (Decided 2026-10-26; replaces the proposed staging → main flow in cicd.md.)
Edge rate limiting Per-IP rate limits are enforced at Cloudflare’s edge (rate-limiting rules), so floods never reach our code: one on api.router.africa (set well above real use) and one on the auth paths of app.router.africa. Configurable in the admin app (screen Edge rate limits): apps/dashboard-api reads and updates the rules through the Cloudflare API with a token limited to this zone’s rate-limiting rules; every change needs a reason and is audit-logged. In addition, apps/api caches rejected key hashes for about 60 seconds, so repeated junk keys don’t each cost a Postgres lookup; valid keys are never cached, so revocation stays immediate (cicd.md ordering rule 6). Plan limits on the number of rules are checked at setup. Per-key limits stay in the gateway (Rate limiting / abuse prevention). (Decided 2026-10-26.)
Database roles (least privilege) Four Postgres roles, each with its own Hyperdrive config: api (apps/api: the hot-path tables only, no Better Auth tables), dashboard (apps/dashboard-api: the core schema), consumer (the queue consumer: the logs schema), migrator (schema changes; used only by CI). Grants are written as migrations. A bug or injection in one Worker can then reach only that Worker’s tables. Extends the separate logs connection already decided (Analytics/dashboard read store). (Decided 2026-10-26.)
Gateway management route allow-list Workers reach the gateway’s management API only for what our code needs: create, update and delete a virtual key; read the model list; read usage logs (reconciliation). A small in-cluster proxy (Traefik on internal entrypoints) exposes inference routes on one Kubernetes Service and only those management routes and methods on another, each with its own Workers VPC Service: apps/api binds inference only; apps/dashboard-api binds management. Everything else (provider keys, configuration, plugins, the dashboard UI) is reachable only from Bifrost’s own dashboard over WireGuard. Bifrost’s admin credential is a Worker secret on apps/dashboard-api only, and a conformance test proves a virtual key alone cannot call a management route. Exact Bifrost paths confirmed against the running instance (backlog #18). (Decided 2026-10-26.)
Kubernetes network policies Default-deny between pods in every namespace, using K3s’s built-in network policy controller. Allowed: cloudflared → the gateway proxy and niobi-proxy; Traefik → niobi-proxy (webhook path); the gateway proxy → Bifrost; Bifrost → its Postgres; each pod’s needed egress (providers, Niobi, Datadog, Cloudflare). (Decided 2026-10-26.)
Cloud account security MFA (authenticator app or hardware key, no SMS) on every account that controls infrastructure: DigitalOcean, Cloudflare, PlanetScale, GitHub, Datadog, Resend and the Niobi portal. In place, per the operator (2026-10-26). API tokens are scoped to the minimum and listed, with who holds them, in the secrets inventory (../infra/secrets-inventory.md). (Decided 2026-10-26.)
Container registry GHCR (GitHub Container Registry), private, for apps/niobi-proxy and any other droplet images. CI pushes with GitHub’s built-in token; K3s pulls with a read-only token held as a Kubernetes secret. A cleanup policy keeps the last ~10 versions so usage stays within the free allowance (limits confirmed at setup). (Decided 2026-10-26.)
Migrations and the staging database Migrations are Drizzle files in packages/db, applied by CI with the migrator role as the first deploy step, before any code (CI/CD pipeline). Staging is a self-hosted Postgres in the K3s cluster on the droplet (revised 2026-10-26; production joined it on 2026-11-01, Ledger database): its own CloudNativePG cluster under infra/postgres/, the same Postgres major version as production, the same roles and migrations, reached by the staging Workers through Hyperdrive over a Workers VPC Service (the private tunnel). Separate from Bifrost’s bundled Postgres. Staging data is disposable and reseeded, never production data. Test databases: PR and CI tests run against a throwaway Postgres container on GitHub’s hosted runners (the self-hosted runner never runs PR code); no PlanetScale branches. Both environments’ migrations run on the self-hosted runner, since both databases are private. Forward-only, expand then contract (cicd.md section 4). (Decided 2026-10-26.)
Admin hosting apps/admin is its own Cloudflare Pages project (separate from the dashboard, so admin code never ships in the customer bundle), behind Cloudflare Access (Admin origin access), deployed by the same pipeline right after the dashboard. (Decided 2026-10-26.)
Configuration and secrets management Non-secret configuration (URLs, feature flags, limits) lives in the repo: the [env.staging] / [env.production] blocks of wrangler.toml and values-staging.yaml / values-production.yaml for Helm, deployed by the pipeline. Secrets: the master copy of every secret, per environment, is a Bitwarden vault (free plan), which also holds account MFA recovery codes, WireGuard keys and the off-server copy of Bifrost’s encryption key. From there they are set as Worker secrets (wrangler secret put), Kubernetes Secrets (kubectl, over WireGuard) and GitHub Actions repository secrets, the last limited to what CI needs and prefixed STAGING_ / PROD_ only where a value differs per environment (GitHub Free has no per-environment secrets for private repos). Today none does: CI holds one Cloudflare deploy token (secret CLOUDFLARE_API_TOKEN) and the account id (variable CLOUDFLARE_ACCOUNT_ID), and no database credentials. Secrets never enter the repo. ../infra/secrets-inventory.md lists every secret by name, where it is deployed, who holds it and how to rotate it; it never holds values. Upgrade path when a second person needs access: Bitwarden Secrets Manager (free tier), read directly by CI. Rancher stays deferred: it is a cluster UI, not a secrets store; kubectl, helm and k9s over WireGuard are enough for one node and one operator. (Decided 2026-10-26.)
Org suspension Staff can suspend and reactivate an org (admin screen 38), with a reason. organizations.status (active / suspended) is read on the request path with the rest of the org, at no extra cost. While suspended: every API key in the org is refused with a specific “account suspended” error from the error catalogue; members can still sign in, but the dashboard is read-only with a notice; top-ups and transfers are blocked. Reactivating restores everything; nothing is deleted. Suspending a user (screen 39) uses Better Auth’s ban: their sessions are revoked and they can’t sign in, but keys they created keep working, because keys belong to the workspace; staff revoke those separately if needed. Freezing a workspace (screen 40) is unchanged: its keys fail auth. Suspend, reactivate, ban and freeze are all audit-logged. (Decided 2026-10-26.)
Developer docs site A custom docs app in the website repo (Astro, our own layouts and design system) at router.africa/docs, replacing the earlier Starlight choice, built to match the features of OpenRouter’s docs: llms.txt and llms-full.txt, a Markdown version of every page, “Copy page” / “Open in ChatGPT / Claude”, language and SDK tabs, ⌘K search, callouts, dark theme, changelog. API reference renders from our own curated OpenAPI spec, hand-maintained next to apps/api and checked against the real API by a contract test; this replaces “no generated reference” while keeping “documented means supported”, since nothing enters the spec without a decision. Request Builder generates code only and never sends requests (no CORS on api.router.africa; keys never go into a web page). Ask AI is a nice-to-have, not launch-blocking, with a cost cap. Plan: ../execution/track-d-developer-docs.md. (Decided 2026-10-26.)
API error contract and response extras Error format: OpenRouter’s envelope, {"error": {"code", "message", "metadata": {"error_type", ...}}}, so ported client code keeps working, using only the error_type values we actually emit. The full catalogue is api-errors.md (written 2026-10-26): each error also carries metadata.reason for the specific cause; out of credits and spend caps are 402; no x-ratelimit-* headers, Retry-After only. No cost or balance in the response body at launch: completions keep OpenAI’s exact shape; balance and key usage come from GET /v1/key. Request logs store the client’s country and ASN (from Cloudflare’s CF-IPCountry and cf.asn) but not the IP address. Logging the IP is deferred; a nullable client_ip column is created now and never written, so turning it on later needs no migration on the largest table (future-optimizations.md). (Decided 2026-10-26; closes api-surface.md section 6 items 2 and 3, and api-key-security.md 7.2.)
FX markup FX rates are stored as the provider reports them (fx_rates: official and parallel; FX providers and rate basis, revised 2026-11-02), and a separate FX markup (a percentage, global or per currency, configured in admin as a markup rule with scope fx) is applied wherever FX is used, always on the side that favours us, like a bank spread. Model pricing: cost in USD × FX rate (the currency’s basis rate) × (1 + FX markup), then the token markup (Markup configuration). Credit moves: each leg through USD is marked against the customer, so the destination receives mid ÷ (1 + markup on the first leg) ÷ (1 + markup on the second leg); a USD leg has no FX markup. Between two non-USD currencies that is two markups, accepted because moves are rare, staff-run and visible before confirming. A missing FX markup rule is an error, never a silent zero: pricing without it is unsafe. Revises Price source of truth / repricing cadence, which used plain mid-market FX. (Decided 2026-10-26.)
Credit moves are support-only Customers don’t move credits between workspaces. The customer Transfer screen and buttons are removed (build inventory screens 16, 18, 23, 24). On request, staff use Move credits on the admin Workspace detail screen (40): pick source and destination workspaces, a reason is required, the source is debited and the destination credited in one transaction (transfer_out / transfer_in ledger entries plus a transfers row), at the FX-marked-up rate when the currencies differ (FX markup), audit-logged. The separate transfer markup rule is dropped. A workspace can be deleted only at a zero balance; support moves any remainder first. Replaces the build inventory’s Credit transfer and Cross-currency transfer features. (Decided 2026-10-26.)
Operator answers to the 2026-11-01 day run Answers to the run’s 14 questions (execution/runs/2026-11-01-your-actions.md), decided 2026-11-01. (1) Charts use Recharts. (2) API key base62 alphabet order is 0-9A-Za-z, fixed for good. (3) At sign-up, USD is always the fallback currency when the country’s currency isn’t enabled. (4) Members may read their org’s member list and create their own keys in workspaces they belong to. (5) The rate-limit audit event is named the same in every document: rate_limit.changed (the task text is corrected to match db-schema.md). (6) Customers see all their request history since the account began; no date limit in the UI (metadata is kept indefinitely, as Analytics/dashboard read store says). (7) Top-ups for now: the customer enters an amount, we create a hosted checkout for that amount through Niobi’s API, and show it in an iframe inside the dashboard. Minimum and maximum amounts come from Niobi’s documentation. (Hosted checkout and iframe superseded 2026-11-02: see Top-ups through Niobi’s direct collections API*.)* (8) Unbilled usage is charged automatically. If the workspace has no balance, the charge stays pending and shows on the admin screen. Once the customer tops up, it’s settled on their next request. Every charge uses the rate in force when the request was processed, never today’s rate. (9) Margin is computed per request at the FX rate and markup already baked into that request’s published price (the rate at that moment), never the latest FX rate. (10) A max_tokens above a model’s ceiling is capped; 400 max_tokens_above_limit is removed from api-errors.md. (11) Idempotency windows: 30 s automatic, 24 h with a client key. Hold fallback when a model has no default or ceiling: 100,000 output tokens. Stored content: 1 MiB per request, past which the stored copy is cut and marked. (12) The consumer database role gets UPDATE on core.api_keys.last_used_at only (P1.06). (13) FX source for the markets the free source doesn’t cover: the Mansa API (mansaapi.com; design in FX providers and rate basis). (14) The dashboard-api test suite also runs on real Postgres, but not on every PR: only on deploys to staging (main) and production, plus manual runs.
Daily backups; no staging Bifrost (2026-11-02) Operator decisions while preparing P0.08. (1) Production app database: backed up once a day, a pg_dump to router-africa-db-backups-production (production/daily/), kept 30 days by an R2 lifecycle rule. This replaces the continuous WAL archiving and point-in-time recovery in Ledger database. Accepted: up to a day of data can be lost if the droplet is lost. (2) Staging has no Bifrost. Staging’s Workers use a mock gateway (canned completions with usage, in-Worker credential management), so there is no second Bifrost release to maintain on the droplet. Accepted: staging never exercises the real gateway. The real Bifrost path (P3.01 adapter, P0.13 checks) is verified against production Bifrost with a low-limit key before launch. P0.11 (a provider key on staging Bifrost) no longer applies.
Route split in nginx; database links without certificate checks (2026-11-02) Operator decisions while installing P0.10. (1) The gateway allow-list (Gateway management route allow-list) runs in a small nginx in front of Bifrost (infra/gateways/bifrost-1/route-split), not on extra Traefik entrypoints. Why: K3s’s Traefik also serves production Bifrost’s public address, and reconfiguring it risks production. K3s maintains Traefik itself, while the nginx is one short config with no state. Allowed: on inference, POST /v1/chat/completions and GET /health; on management, GET|POST /api/governance/virtual-keys, GET|PUT|DELETE /api/governance/virtual-keys/{id}, GET /api/models and GET /api/logs. Added 2026-11-02 (P3.01, code #70, merged): GET /api/models/details, the only route with per-token costs; live since 2026-11-05 (as-built section 16). Revisit at the pre-launch lockdown: once Bifrost’s public ingress is removed and Traefik serves only the Niobi webhook path, either keep both or move that path to nginx and disable Traefik. (2) The Postgres Workers VPC Services run with certificate verification disabled. CloudNativePG’s certificates come from a private CA, and Workers VPC can’t take a custom CA yet. Traffic is encrypted by the tunnel. Follow-up: a Cloudflare Origin CA certificate on the databases, then full verification.
Admin spend cap period: monthly (2026-11-02) The admin-set cap (Spend cap authority: a Bifrost virtual key budget in USD) resets monthly, Bifrost reset_duration: "1M". This is the default in the gateway adapter (packages/gateway/src/bifrost.ts, budgetPeriod, P3.01, #71). Confirmed by the operator 2026-11-02.
Operator decisions, 2026-11-02 (afternoon) (1) Droplet stays at 4 GB (2 vCPU, 70 GB disk). No swap or resize; P0.09 (self-hosted runner) no longer waits on memory. (2) One Turnstile widget for staging and production is accepted (the reviewer flagged the shared widget in #37). (3) Resend: the router.africa sending domain is verified. (4) Datadog is deferred; observability (P9.07) picks it up later. (5) Niobi: the operator allowed the hosted checkout in an iframe, but Niobi’s pages refuse framing, so top-ups moved to the direct collections API (Top-ups through Niobi’s direct collections API). Webhook source IPs, signing and top-up limits come from Niobi’s public docs (docs.niobi.co); the operator supplies the API credentials.
Operator decisions, 2026-11-07 (developer docs) (1) Theme: the docs are dark and light, following the device setting by default, with a toggle, matching the marketing site. (2) Request Builder models are a fixed list maintained in the docs, not fetched from staging. (3) One PR for the Track D website work (D.02 mockups through D.08), reviewed visually before merge. D.10 (Ask AI) stays out of the current plan.
FX providers and rate basis (2026-11-02) Detail for answer 13 of the 2026-11-01 run (Operator answers to the 2026-11-01 day run). (1) More than one provider. No single source covers every currency, so rates come from several: Mansa API (mansaapi.com, GET /api/v1/markets/forex) for the African currencies it covers, Frankfurter for the ones it covers, and manual entry by staff for the rest (at launch SLE, which Mansa doesn’t cover). Each currency has its own provider order, set in admin, and the job tries them in that order. Every fx_rates row records its provider and provider_fetched_at. (2) Both rates are recorded. Each row stores the official rate and, where the provider has one, the parallel (P2P, “black market”) rate, all as local currency per 1 USD. Columns are named from our side, never with the provider’s labels: parallel_usd_cost is what we pay for 1 USD, parallel_usd_proceeds is what we’d get for selling 1 USD. Each provider’s adapter maps its own field names onto these columns in one place, with a test. For Mansa the mapping is confirmed against a real response before it ships (from its public board, its “Sell” column is our cost). (3) Rate basis per currency. currencies.fx_rate_basis is official (default) or parallel_usd_cost, set in admin. Customer money ends up converted into USD, so on a parallel basis the price uses what we pay for a dollar. A currency moves to the parallel basis only after Niobi confirms, for that market, the rate it actually converts at and that pricing off the parallel rate is acceptable to the regulator. (4) Daily value. Mansa is fetched hourly (24 calls a day, inside its free tier of 100), and the day’s rate is the median of that day’s samples for the basis. (5) Bad quotes are refused. A parallel quote where cost is below proceeds (Mansa’s ZMW on 2026-11-02) is not stored: the official rate still is, an alert fires, and a currency on the parallel basis falls back to official for that day and is flagged on admin screen 44. A currency on the parallel basis with no parallel rate from any provider is handled the same way. (6) FX markup still applies (FX markup). Where the basis is the parallel cost the spread is already in the rate, so that currency’s FX markup rule is expected to be 0%; the rule must still exist, since a missing rule is an error. Revises FX markup (“stored at mid-market”) and Billing/pricing model (Frankfurter as the only source).
Top-ups through Niobi’s direct collections API (2026-11-02) Supersedes Q7 of the 2026-11-01 answers (hosted checkout in an iframe). Top-ups use Niobi’s direct collections API (/api/v4/niobi-unified-collections) with our own top-up screen (17). Why: Niobi’s hosted checkout can’t be framed (X-Frame-Options: SAMEORIGIN on every checkout page, checked 2026-11-02), and payment links can’t carry our reference, while direct collections take our top-up id as third_party_reference_1. Most markets complete on our page (phone prompt, Nigeria bank transfer, Côte d’Ivoire Orange OTP); Wave and South Africa open the provider’s page in a new tab. Given up: card payments, which only hosted checkout offers; the likely route for cards later is a payment link in a new tab. Malawi has no collections yet, so no top-ups there until Niobi adds it. Staging uses Niobi’s sandbox with sandbox credentials; automated tests use niobi-proxy’s mock. Webhook source IP [redacted-ip] (sandbox and production); the droplet’s [redacted-ip] is whitelisted with Niobi in both. Scope, flows, methods and limits: niobi-integration.md. (Decided 2026-11-02.)
Code review fixes, P0–P4 (2026-11-01) The reviewer (minaamfasihi) pushed fixes as commits on the PRs, not as comments. They are in main through #4–#10, #12, #13, #17, #23, #37, #45, #56, #58, #59, #61 and #21. Changes to pricing, schema and auth are written into pricing-model.md (significant-figure rounding, refused inputs, stale FX), db-schema.md (rate_sig_figs, fx_markup_rule_id, non-negative markups, per-leg transfers, workspace-first rollup keys, keys that keep authenticating while a sync is pending) and auth-security.md 2.1 (one auth pass, a 16 MiB body cap, failed-lookup throttling). The rest is recorded here. Gateway and event contracts (P1.09): the chat body sent to the gateway is an allowlist of OpenAI fields, not a passthrough: gateway routing and fallback fields are dropped and n is 1. Credentials are an opaque Secret type that never prints. GatewayError.detail is structured and redacted, never the raw response. The usage promise always settles, and a null result releases the hold. Niobi’s status and callback carry the amount and currency collected, and the credit path checks them; topupId is the idempotent merchant reference for collections. Usage-event strings are length-bounded, errorReason comes from the catalogue, and a failure before the model resolves logs UNKNOWN_MODEL_ID. Request path and logs store (P1.10): usage is recorded per queue batch (three statements, sums done in SQL so money stays numeric, rollup groups in lock order). Every Hyperdrive config must have query caching disabled (a cached read could serve a stale balance or key). recordAudit and recordJobRun ignore a conflict on their primary key, so a retry after a write that already landed is a no-op. sync_error and discrepancy details never hold a credential value. An idempotency claim takes over an expired row (ON CONFLICT DO UPDATE … WHERE expires_at < now()), never DO NOTHING. Tenancy (P1.02): membership rows have composite foreign keys, so a workspace member must belong to the workspace’s org; workspace names are unique per org among non-deleted workspaces. Triggers enforce the currency lock after the first top-up and stop an invitation from granting a workspace outside its org. Scheduled jobs (P4.02): logs.job_runs is written through the LogsStore after the core.scheduled_jobs change commits, not in a transaction spanning both schemas, so logs can move to another store; a failed write is retried once, then logged. Dispatch uses Promise.allSettled, so one failing job doesn’t lose the others’ outcomes. Job errors are scrubbed of our keys, bearer tokens and URL passwords before they are stored or logged. Every job must be safe to run again, and to run twice at once (a reclaimed lease, or Run now followed by the schedule). Cron day fields follow Vixie cron semantics. Workers (P0.07): both Workers have workers_dev = false and preview_urls = false, so neither is reachable except through its routes, where the zone’s WAF, rate limits and (for the admin API) Access apply. Staging still shares production’s Turnstile widget and secret. CI and secret scan (P0.12): CI tokens are read-only. Actions are pinned to commits and the gitleaks image by digest. A PR scans only its own commits; pushes and a weekly run scan the full history. The sk-ra- secret rule requires entropy above 3.5, so made-up test keys don’t fail the scan. A newer push cancels a PR’s running CI, and the Turborepo cache is kept between runs. The no-logging-headers lint rule also catches destructured headers, whole request objects and loggers reached through a property. Closed without merging: #11 (P2.06 discount and transfer helpers): “not needed”, because the same commit had already reached main through the stacked #12. #16: replaced by #59, which merged its head.
Code review fixes, 2026-11-02 The reviewer’s second pass, on the stack merged 2026-11-02. Fix commits on #31, #38, #50, #51, #57, #60 and #63, and fix PRs #72–#75, #90 and #91. Spec changes are written into api-key-security.md section 3 (in-place rotation, revoke, who can create workspace keys), cicd.md (where the real-Postgres tests run, pipeline least privilege, Dependabot) and secrets-inventory.md (the Pages preview token). The rest is recorded here. Keys (#31): every key change commits with its audit event (key.created, key.updated with before and after, key.rotated, key.revoked) in one transaction. The key row is locked FOR UPDATE during a change, so revoke and rotate wait rather than race; gateway calls time out after 5 s. A malformed CREDENTIAL_KEY_V1 disables key changes (503), not every route. One spendWindowStart (@router-africa/pricing, #51) serves hold, settle and the key list, so their windows can’t drift. Audit (#72, #75): LogsStore.recordAudit takes the caller’s transaction, so audit events stay atomic with the change through one write path; a store off Postgres would need an outbox written in that transaction. An IP that inet would reject is dropped, so a malformed header can’t roll back the change. Credentials are refused anywhere in metadata or the reason; the reason is capped at 1,000 characters and metadata at 8 KB. transfer.created needs a reason. Every org and workspace change now writes its event (org.updated, org.deleted, workspace.created, workspace.updated, workspace.deleted, workspace.member_added, workspace.member_removed); a no-op writes none. A currency change that loses the race with a first top-up answers 409. Money (#74): staff credit moves use the shared exact-decimal moveRate/convertMove (FX markup: mid via USD, markup on each non-USD leg, credited amount rounded down), write the transfers row with the two ledger entries, lock both balances in id order, refuse a move with no FX rule, and answer 409 when credits were spent between quote and move. Settle’s returned absorbedShortfall is exact, and published rates record fx_markup_rule_id. Pipeline (#38): packages: write only on the image push; the Cloudflare token only on the jobs that deploy to Cloudflare; PR previews use their own Pages-only token; a production deploy refuses a tag whose commit isn’t on main or release/*; inputs reach shell scripts through env. Droplet (#57, #60): niobi-proxy, the daily backup, route-split and cloudflared run with no Linux capabilities and the default seccomp profile. The route split forwards exactly the path it checked, not the raw request line. Hyperdrive (#63): the runbook creates configs with --caching-disabled; the six existing configs predate that and still cache (as-built section 17). Secret scan (#73): three values in history that aren’t secrets are allowlisted by exact value. Test harness (#90): real-Postgres template databases are built one at a time (roles are server-wide); production runs migrations once and isn’t affected. Dependabot (#91): no automatic major-version updates for npm or GitHub Actions; each major is upgraded on its own branch, and alerts still flag vulnerable versions. Closed as already in main: the screens stack (#14, #20, #25–#30), #18, #19 and #24 reached main through the integration branch (#65); #66 duplicated #64.
Code review fixes, 2026-11-04 and 05 The reviewer’s money and performance review of main, as PRs #95–#105 (merged 2026-11-04 and 05). Merged without CI: GitHub Actions has not started any job since 2026-11-04 (account billing), so these PRs ran only the reviewer’s local tests; the real-Postgres concurrency suite, which #105 asked to have run before merging, has not run against them. Spec changes are written into db-schema.md (lock order, claim_and_hold, usage_charges_daily, holds.window_*, the unbilled index, top-up crediting) and the Hold close-out and Ledger invariant check entries above. The rest is recorded here. Top-ups (#95): the fee is exact (it was float math rounded up, a cent too high in 4 of 72 realistic cases); credit only what was collected; a failed create stays pending. The real Niobi client (P6.01) must return the collected amount and currency. Exact money on every screen (#97, 610bfbf): amounts from numeric(24,8) are formatted and summed as exact decimals, never through Number() (a float shows a 12-digit KES balance wrong), in /v1/key, billing, usage and logs, and admin; only the admin margin percentage becomes a number, after the exact arithmetic. Unbilled charging (#99, #101): closing a still-open hold gives back the key’s spend-window reservation, so a key isn’t refused key_spend_cap_exceeded until the window resets; settle’s balance update also returns whether the workspace has open unbilled usage, so the request path no longer spends a query per request asking (operator answer 8 unchanged: it’s charged on the workspace’s next settled request). Daily rates (#103): FX age is judged by the run’s date, not the wall clock, so a run past midnight or a reclaimed lease uses the right fallback. Hold in one round trip (#105): core.claim_and_hold, two round trips before the gateway call instead of about seven (roughly 0.6 s saved at 100 ms a trip). Stacking slip: #100 merged into #99’s branch after #99 had merged, so it was re-opened as #101; #106 (per-org in-flight cap) did the same and is re-opened as #107, still open.
Per-org in-flight cap (2026-11-05) Code #107 (re-open of #106, which merged into a branch that had already merged). Why: Credit hold/settle mechanism accepts a slight overage on the last request, but an account running many full-context requests at once could absorb many times its balance, and with holds off nothing limited concurrency at all. Decided: organizations.max_in_flight, default 20, counts every workspace and key in the org, holds on or off. It is checked inside core.claim_and_hold under a per-org advisory lock; over it the claim rolls back and the API answers 429 concurrency_limited (api-errors.md), with no Retry-After. Claims older than 15 minutes stop counting, so a dead Worker can’t block an org for good. Staff change the cap per org in admin (“Requests at once”, 1–10,000, reason required, in_flight_cap.changed); an admin org PATCH now changes exactly one setting, so each change gets its own audit event. Worst-case exposure per org is about 20 × the largest single-request overage; raise it per org for batch workloads. The default is a judgment call, open to tuning once real traffic shows typical concurrency.
Code review fixes, 2026-11-05 (afternoon) The reviewer’s next pass on main, as PRs #113, #114, #115 and #117 (minaamfasihi, merged 2026-11-05). Migrations 0017–0019; details in db-schema.md sections 4.4, 5 and 8.3. (1) Settle and release are one database call each (#113, core.settle, core.release_request): same order, locks and results, but the balance row is locked only while the function runs, which lifts the per-workspace settle ceiling from about 3 a second. (2) A dead Worker’s idempotency claim is taken over after 15 minutes (#113), the same age at which the in-flight cap stops counting it; before, a client Idempotency-Key stayed blocked for 24 h. (3) Bad gateway token counts (negative, fractional, huge) now count as no usage: the hold is released and the usage reconciliation charges from the gateway’s logs (#113). (4) Top-up status is asked of the provider at most once per 10 seconds per top-up (#114), however many tabs poll; staff “check now” always asks. The scheduled pending-top-up job (Top-up status check) is still phase 6. (5) The ledger invariant check is incremental (#115): it reads only entries since its last checkpoint, and fully recounts every 7 days and on the first run. (6) Expired idempotency claims, sessions and verifications are purged every 10 minutes by a new expiry_purge job (#117), not daily. (7) Staff credit changes are exact and happen once (#117): adjust and move amounts allow at most 8 decimals and 12 integer digits, matching numeric(24,8), with zero checked exactly; each dialog sends an operationId used as the ledger entry or transfer id, so a double click or retry applies once (alreadyDone: true) and a reused id with a different change is refused (409 OPERATION_REUSED). (8) Idempotency is per API key (#117): the key is part of the hash, so two keys in one workspace sending the same body each run and are each charged. At the deploy, a client retrying with an Idempotency-Key sent before it runs again once. (9) Settle completes the idempotency claim in the same call (#117): 3 database calls per request instead of 4.
Operator decisions, 2026-11-07 (API contract) (1) Content retention: not saved by default; saving is opt-in. Supersedes the 2026-09-20 “saved by default, opt-out” decision in Audit trail & prompt/response retention (corrected there); matches Terms 8.4, Privacy 3.3/8 and the website copy. The workspace flag content_logging_enabled defaults to false. (2) max_tokens above the model’s ceiling is silently lowered to the ceiling, not an error; max_tokens_above_limit (and the max_tokens_exceeded type) is removed from api-errors.md, the spec and the code. (3) POST /v1/responses ships at launch (OpenAI Responses format, streaming included). (4) POST /v1/messages accepts x-api-key as well as Authorization: Bearer, so moving from the Anthropic SDK is a one-line base-URL change; this applies to that route only, and every other route stays Bearer-only. (5) POST /v1/messages errors use Anthropic’s format {"type":"error","error":{"type","message","reason"}}, with our reason kept inside error; the status-to-type mapping and the mid-stream event: error are in api-errors.md section 7.

Item Why it’s unresolved Blocks
None All open decisions are closed as of 2026-09-20. Deferred work is tracked in future-optimizations.md and launch follow-ups in the backlog (item 17). —

Deferred optimizations and the hooks that keep them cheap are tracked in future-optimizations.md; add a row there whenever a decision says “later”.

Keep this file current as decisions close — an item moving from Open to Decided should be a deliberate edit, not something that quietly happens elsewhere and never gets reflected here.