Skip to content

Router.Africa — API Key Security

Backlog item #15: keeping API keys from leaking, and limiting the damage when one does. Most of the mechanism is already decided and specified; this document ties it together as a key lifecycle, states what a leaked key can and can’t do, and adds what was missing: the Bifrost virtual key’s secret, leak detection and the response plan. Sources: decisions-log.md (API key security model, Public API authentication, API key ↔ Bifrost virtual key mapping, Key ↔ Bifrost sync and failure path, Spend cap authority, Rate limiting) and auth-security.md (key format, authentication, the credential-plane rules, the conformance suite). Where they disagree, the decisions log wins. Anything marked proposed is not decided; section 10 collects them.

Data residency, KYC and AML are out of scope (handled through Niobi’s signature mechanism).


Concern Control Specified in
Generation sk-ra- + 37 random base62 characters (about 220 bits) + a 6-character CRC32 checksum. Shown once auth-security.md 2.2
Storage SHA-256 hash only; lookup by hash. Raw key never stored. key_prefix (first 12 characters) for display db-schema.md 4.2, auth-security.md 2.2
Authentication Authorization: Bearer only. One identical 401 for unknown, revoked, expired and pending keys, key masked in the message. No cookie is ever read auth-security.md 2.1 to 2.3
Scoping Model list, admin cap, user cap, rate limit, expiry, ownership Section 4
Creation and sync pending row, then Bifrost virtual key, then active, then show the key. No key is shown until both sides exist Decisions log, Key ↔ Bifrost sync
Revocation Marked revoked in Postgres first (dead immediately), then deleted in Bifrost with retry Section 3
Transmission HTTPS only; key never in a URL, never in web storage, never in a log Section 6
Detection and response Anomaly signals, customer notice, revoke, leak scanning Section 7
Verification Conformance suite, plus the lifecycle tests in section 9 auth-security.md 6

A key is a bearer credential for one workspace’s model spend. Nothing else.

A leaked key can Bounded by
Spend the workspace’s credits on models the key allows The workspace balance (the hold path refuses when it runs out), the key’s allowed_models, the user-set cap (credits, in Postgres), the admin-set cap (USD, in Bifrost) and the rate limit (in Bifrost, effectively per key)
Keep working until revoked or expired Revocation is immediate (section 3); expires_at limits the window if set
Run up usage faster than a person would notice Detection is asynchronous (section 7). The real-time backstops are the caps and the rate limit
A leaked key can’t Why
Read logs, stored prompts or responses, balances or billing The dashboard and admin planes use a session cookie only; an API key sent there gets 401 (auth-security.md rule R2)
Create, list or revoke keys, top up, or transfer credits Same: those are dashboard routes
Reach another workspace The key alone determines the workspace; OpenAI-Organization and OpenAI-Project headers are ignored
Exceed the workspace balance by much Two-phase hold; a shortfall on the last request is the accepted risk (decisions log, Credit hold/settle mechanism)

The exposure is therefore money in the workspace balance, on the allowed models, at the key’s rate limit. That is why per-key caps and short-lived keys matter (section 4).

State Meaning Authenticates?
pending Row exists, Bifrost virtual key not yet confirmed. Never shown to the customer No
active Both sides exist. Shown once at this point Yes, until expires_at
revoked Marked in Postgres, then the Bifrost key is deleted with retry (sync_state pending_delete until confirmed) No
  • Create: the ordered flow with compensation (decisions log). A crash between steps leaves no usable half-key. The raw key is held in memory only while the user is on the screen (auth-security.md R5).
  • Rotate: in place, on an active key only (screens 14 and 15). The row and its gateway credential stay; the key gets a new hash, so the old value stops working at once and the new one is shown once. There is no grace period. For a cutover without downtime, the customer creates a second key, moves their systems to it, then revokes the first; the overlap is the grace period, and expires_at on the old key makes a planned cutover safe. The docs should say both plainly. (As built, code #31, 2026-11-02.)
  • Revoke: takes effect on the next request, because apps/api reads the key from Postgres on every request. Do not add a cache in front of that lookup without a bounded lifetime and a way to invalidate on revoke; a cached key would stay usable after revoke. The revoking transaction also wipes the stored gateway credential (secret_enc, enc_version), whatever the gateway does, so pending_delete keeps only the reference. If the Bifrost delete fails, the row stays pending_delete, the key is already dead at the edge, and the discrepancy view lists it as revoked_still_live until cleaned up. A repeat revoke writes no event and only retries a pending delete.
  • Who can act: a customer’s Owner and Admin roles (step-up required to create, rotate or revoke, auth-security.md 3.2). Members create only their own keys, in workspaces they belong to (decisions log, Operator answers to the 2026-11-01 day run, answer 4), never workspace-owned ones, since they couldn’t manage them. Staff can revoke from the admin app. Every create, update, rotate and revoke is audit-logged in the same transaction as the change (key.created, key.updated with before and after, key.rotated, key.revoked).
  • Expiry: optional per key, enforced at authentication. The same 401 as any bad key.
  • Ownership: a key is workspace-owned or belongs to a user (owner_user_id). Removing a member does not silently delete their keys; the removal flow should prompt the admin to revoke or reassign them (proposed).

Each control has one owner, so nothing is enforced twice or in conflict.

Control Where it lives Enforced by Set by
allowed_models per key api_key_models apps/api (before Bifrost) and Bifrost’s allowlist (second line) Customer, per key
User-set spend cap (credits, with a period) api_keys.user_spend_cap, key_spend_windows Postgres, in the hold transaction Customer
Admin-set spend cap (USD) api_keys.admin_spend_cap_usd Bifrost (virtual key budget) Staff. A fixed backstop that doesn’t follow FX
Rate limit Org setting, copied to each key’s virtual key Bifrost Staff. Per virtual key only, so an org with many keys can exceed it in aggregate (accepted)
Expiry api_keys.expires_at apps/api at authentication Customer
Workspace The key resolves to exactly one apps/api Fixed at creation

Guidance for customers (docs and the dashboard’s key screen): one key per system or environment, a model list of only what that system needs, a spend cap, and an expiry for anything temporary. The default for a new key is the org’s default model list with no user cap, so the dashboard should nudge toward setting one (proposed).

5. Gap: the Bifrost virtual key’s secret and direct access

Section titled “5. Gap: the Bifrost virtual key’s secret and direct access”

Resolved 2026-10-25. Storage: encrypted at rest (decisions log, Bifrost virtual key secret storage). Direct access: Workers reach Bifrost through a Workers VPC binding with no public hostname, and the droplet is locked down before launch (decisions log, Private service access and Droplet network lockdown). The analysis below is kept for the reasoning.

This is not covered in the decisions log or the schema, and it matters more than anything else in this document.

What we know. Each Router.Africa key maps 1:1 to a Bifrost virtual key, and the Worker “calls Bifrost using the mapped virtual key” (backend-scope.md section 3). The schema stores only bifrost_vk_id, an identifier, in api_keys. To make the call, the Worker needs the virtual key’s secret value (auth-security.md 2.1 says it goes in a header such as x-bf-vk).

Two questions with no written answer:

  1. Where does the virtual key’s secret value live? It must be readable by apps/api on every request, and it is as sensitive as the customer’s own key, because it authenticates to Bifrost. If it sits in a plain Postgres column, a database leak or a read-only query hands out working Bifrost credentials for every customer. It should be encrypted at rest with a key held only as a Worker secret (proposed), and never logged or returned by any route. Whether Bifrost returns the value at creation, and whether it can be re-read later, is to verify against the running instance (the backlog #18 verification already covers Bifrost’s key fields).
  2. Can anyone reach Bifrost directly? backend-scope.md describes Bifrost as private, but the install as built (installation-as-built.md) exposes it on a public hostname (prod-bifrost-core.router.africa) through Traefik and Let’s Encrypt, with enforceAuthOnInference: true. If a virtual key value ever leaks (through the database, a log or a trace), it can be used straight against Bifrost, skipping the hold, the settle and the ledger entirely. The only thing that would notice is the hourly usage reconciliation, as a not_charged row, after the fact.

Proposed fix (needs a decision): make Bifrost unreachable from the internet for inference. Put it behind a Cloudflare Tunnel with a service token or mTLS, the same pattern already chosen for apps/niobi-proxy, so only our Workers can call it; keep the public hostname only for the admin UI if it’s needed, behind Cloudflare Access. Combined with encrypted storage of the value, a leaked virtual key is then useless without also being inside our network path. If the public route must stay, at minimum restrict inference paths at the ingress by a shared secret header that only apps/api sends, and alert on any Bifrost request whose id isn’t ours (the usage reconciliation already finds these).

  • HTTPS only. HSTS with includeSubDomains on router.africa (auth-security.md 3.4). For api.router.africa, plain HTTP should be refused, not redirected: a redirect means the key has already crossed the network in clear text on the first request (proposed; a Cloudflare rule on port 80 for that hostname).
  • Never in a URL. A key in the query string or body returns 401 (auth-security.md 2.1, conformance test 3), so it can’t land in access logs, referrers or browser history.
  • Never in a log or trace. Authorization, Cookie, Set-Cookie and x-api-key are stripped from Datadog, Workers logs, error reports and queue messages; a lint rule forbids logging headers objects; conformance test 20 fails the build if the test key appears in any output. The request log stores key_id, never the key, and the audit trail stores no secrets.
  • Never echoed. Errors show the masked key (sk-ra-…abcd) only.
  • Never in browser storage. The full key exists only in memory on the “created” screen (auth-security.md R5). The keys list shows the prefix and last-used time.
  • Customer guidance. Docs tell developers to keep keys in environment variables or a secrets manager, never in front-end code (there is no CORS, so a browser call with a key fails by design), and never in a public repository (see 7.1).

Enforcement in real time is the caps and the rate limit. Detection below is asynchronous, so it alerts and informs; it does not block (the decisions log rules out reading real-time limits from the async log).

7.1 Prevention: leaked in a public place (proposed, after launch)

Section titled “7.1 Prevention: leaked in a public place (proposed, after launch)”

The key format was designed for this: a distinctive prefix and a checksum let a scanner match sk-ra- strings precisely with almost no false positives (auth-security.md 2.2). Register the format with GitHub’s secret scanning partner programme (and any similar scheme) so a key committed to a public repository is reported to an endpoint of ours. On a report, apps/dashboard-api verifies the signature, looks the key up by hash, revokes it and emails the workspace owners. Programme terms and the endpoint’s requirements are to check when we apply; not a launch dependency.

7.2 Detection signals (per key, from usage_hourly and the request log)

Section titled “7.2 Detection signals (per key, from usage_hourly and the request log)”
Signal Suggests
Spend or request rate far above the key’s trailing baseline (e.g. 5x the 7-day hourly average, above a floor) Key in active misuse
First use of a model the key has never called, or a sudden switch to the most expensive model A different user of the key
A dormant key (last_used_at older than 30 days) suddenly active An old leaked key
A sustained burst of 429 at the rate limit for a key that never hit it before Automated abuse
Balance draining much faster than the workspace’s history Any of the above at workspace level
401 flood against one key prefix Someone guessing near a real key (the checksum keeps this off the database)

Thresholds are tuned after real traffic, not guessed now. They feed the P2 alerts in observability.md section 5.7 and are visible to staff in the admin app.

Gap: no client location on requests. The request log has no client IP, country or ASN column (db-schema.md, logs.request_logs), so “used from a new place” cannot be detected. Adding one is a privacy trade-off (an IP is personal data; a country or ASN from Cloudflare’s CF-IPCountry and cf.asn is much lighter). Proposed: store country and ASN only, on the request-log row, and decide before the schema is final. It’s cheap now and a migration on the largest table later.

  1. Alert: a P2 to the team with the workspace, key prefix and the signal.
  2. Tell the customer: email the workspace Owners and Admins through Resend: “unusual activity on key sk-ra-…abcd”, with the numbers and a link to revoke. (An automatic version is proposed; at launch it is a staff action.)
  3. Revoke: the customer revokes the key themselves (step-up), or staff do it from the admin app, audit-logged with the actor.
  4. Automatic suspension is not proposed for launch. A false positive stops a paying customer’s production traffic, which costs more than the abuse it prevents at launch scale. Revisit with real data.
  5. After: review usage in the exposed window (the request log is the record), and follow the credit policy in section 10.

Runbook (to write, listed in observability.md section 9): a leaked-key report from a customer or a scanner: identify the key by hash or prefix, confirm the workspace, revoke, check the usage window for spend and models, check for other keys created by the same actor in the same window, contact the customer, record the decision on credits.

Customers’ keys are half of it. The other half is what would let someone spend without a key or read what keys protect. The full list of secrets, with where each is deployed and how to rotate it, is ../infra/secrets-inventory.md; the table below is the subset that matters for keys:

Secret Held in If exposed Rotation
Bifrost virtual key values (section 5) api_key_gateway_credentials.secret_enc, AES-256-GCM; encrypted by apps/dashboard-api, decrypted by apps/api Free model spend if Bifrost is reachable (not from the internet once locked down) Re-create the virtual key through the key sync path
VK encryption key Worker secret on apps/api and apps/dashboard-api, versioned Every stored virtual key value can be decrypted, given a database copy New version; re-encrypt rows in the background
niobi-proxy HMAC secret Worker secret on apps/dashboard-api; Kubernetes secret on the droplet Calls to niobi-proxy forged, but only from inside the VPC path Rotate both together
Bifrost encryption key (bifrost-encryption-key) Kubernetes secret, copy kept off the server Bifrost’s stored provider keys can be decrypted Rotate with a Bifrost maintenance window
Provider API keys (OpenAI, Anthropic, Google, Fireworks) Bifrost Someone else spends our provider account Rotate at the provider; update Bifrost
BETTER_AUTH_SECRET_APP, BETTER_AUTH_SECRET_ADMIN Worker secrets, per environment Forged session cookies (a session row is still needed, so limited) Rotate; signs everyone out
Niobi credentials and signing material apps/niobi-proxy only Payment requests signed as us Per Niobi’s process
Resend API key Worker secret Email sent as us Rotate at Resend

See cicd.md section 5 for how secrets are separated per environment.

The auth conformance suite (auth-security.md section 6) already covers key forms, identical 401s, masking, the Authorization header not reaching Bifrost, and no key in logs. Add these key-lifecycle tests to the same suite:

  1. A pending key never authenticates, even if the row exists.
  2. After revoke, the next request returns 401, whether or not the Bifrost delete succeeded.
  3. An expired key returns the same 401 as any bad key.
  4. A key over its user cap, or with the model not in its list, is rejected before Bifrost is called.
  5. Key creation with a Bifrost failure leaves no active row and no shown key.
  6. A stored virtual key value is not readable from any route, response, log or trace, and is not present in the request log.
  7. A dashboard route and an admin route both reject a valid API key (rule R2), and api. rejects a session cookie (R1).
  8. If the leak-scanning endpoint is built (7.1): a report with a bad signature does nothing; a valid one revokes and notifies.
  9. key.created, key.revoked and every scoping change write an audit event with the actor.
  • The Bifrost virtual key secret (section 5) — decided 2026-10-25; see the decisions log.
  • Credits consumed through a leaked key. Credits are non-refundable by design (decisions log, Credit balance & non-refundability), but there is no written policy for a customer whose key was stolen: absorb the cost, refund as a goodwill adjustment (ledger.adjusted, reason required), or none. A business decision, needed before launch so support isn’t improvising.
  • Country and ASN on the request log (section 7.2) — decided 2026-10-26: country and ASN stored; IP deferred (decisions log, API error contract and response extras).
  • Leak-scanning registration (7.1): worth doing after launch; confirm the programme’s requirements.
  • HTTP refusal on api.router.africa (section 6) and what Cloudflare rule implements it.
  • Member removal: prompt to revoke or reassign that member’s keys (section 3).
  • A nudge to set a spend cap when a key is created (section 4).
  • Detection thresholds (7.2): tune against real traffic.