Skip to content

Phase 3 — Request Path (`apps/api`)

Goal: the OpenAI-compatible API, end to end: authenticate the key, gate the model, claim idempotency and hold credit, call the gateway privately, settle, then log off the hot path.

Milestone M1: in staging, a real chat completion with a hand-seeded workspace, key and balance is charged exactly tokens × the locked rate × (1 − discount). The hold closes, the ledger balances, and the concurrency test passes.

This is the money path. P3.09 (the concurrency test) is written alongside P3.04 and P3.06, not after them.

ID Task Who Needs Status
P3.01 Bifrost adapter: inference Agent P1.09, P0.10, P0.13 merged — code #71 (inference and management; against a scripted Bifrost v2.2.3; live checks wait on P0.13, then the five checks in #71’s description)
P3.02 Key authentication Agent P1.10, P1.08 merged — code #23, fix code #61
P3.03 Model gate and request read Agent P3.02, P2.05 merged — code #36 (Bifrost adapter code #71 in production, mock gateway code #69 in staging)
P3.04 Idempotency claim and hold Agent P3.03, P2.06 merged — code #36; hold fallback of 100,000 output tokens code #50; hold as one function core.claim_and_hold code #105
P3.05 Gateway call Agent P3.01, P3.04 merged — code #36 (Bifrost adapter code #71 in production, mock gateway code #69 in staging)
P3.06 Settle and release Agent P3.05 merged — code #36; exact absorbedShortfall code #74; close the hold first, one lock order code #96; open unbilled usage checked inside settle code #101
P3.07 After the response Agent P3.06 merged — code #36 (Bifrost adapter code #71 in production, mock gateway code #69 in staging)
P3.08 Endpoints Agent P3.07 merged — code #36 (chat, models, key; /v1/messages and /v1/responses are launch endpoints, built after P0.13)
P3.09 Hold/settle concurrency test Agent P3.04, P3.06 merged — code #40; runs on real Postgres in the deploy gates and manual Gates runs (code #54)
P3.10 API conformance tests Agent P3.08 local part merged — code #46 (OpenAI SDK, conformance item 20); the staging part waits for a deployed staging stack
P3.11 M1: first charged request in staging You + agent P3.08, P3.09 todo
P3.12 GET /v1/generation (request lookup) Agent P3.08 todo

Who: Agent · Needs: P1.09, P0.10, P0.13

Implement the adapter’s chat completion, streaming and not, through the inference VPC binding:

  • send the decrypted virtual key and our x-request-id

  • set the Worker’s timeout comfortably above Bifrost’s worst-case fallback time

  • do no Worker-level retry

  • map Bifrost errors to the catalogue from P1.08

  • strip every Bifrost-identifying response header

  • Spec: decisions log, Worker → Bifrost timeout/retry policy, Gateway abstraction, Private service access; ../architecture/backend-scope.md section 3 step 4

  • Done when: against staging, a streamed and a non-streamed completion return correctly, no x-bf-* or other gateway header reaches the client, and a provider failure maps to a catalogue error.

Who: Agent · Needs: P1.10, P1.08

Hono middleware:

  • Authorization: Bearer only (plus x-api-key on POST /v1/messages alone): key format and checksum checked locally first, so malformed keys never hit the database.

  • Lookup: SHA-256 hash, then the key must be active and unexpired, its workspace must not be frozen or deleted, and its org must not be suspended (the “account suspended” error); resolve the workspace and org.

  • Responses: identical 401s for every failure, with the masked key only in messages. The key is never logged.

  • Rejected-key cache: about 60 seconds, rejected hashes only, so revocation stays immediate.

  • Spec: ../architecture/auth-security.md sections 1 and 2; ../architecture/api-key-security.md section 6; decisions log, Edge rate limiting, Org suspension

  • Done when:

    • the auth items of the conformance suite that apply to apps/api pass
    • a revoked key fails on its next request
    • keys in a frozen workspace and a suspended org are refused on their next request
    • a burst of the same bad key hits Postgres once per cache window

Who: Agent · Needs: P3.02, P2.05

The single unlocked read:

  • the key’s allowed models
  • the model is on, with a current rate in the workspace’s currency
  • the discount
  • hold settings
  • the content-logging flag

Apply the max_tokens ceiling and default.

  • Spec: ../architecture/db-schema.md section 5 step 1; decisions log, Model availability enforcement
  • Done when: a model not on the key, a model that is off, and a model with no rate are each rejected with no database write. The read uses only primary-key or unique lookups.

Who: Agent · Needs: P3.03, P2.06

One statement, core.claim_and_hold (since code #105, 2026-11-05; it was one short client-side transaction), in this lock order:

  1. Idempotency claim: automatic hash(workspace + exact body), plus the optional Idempotency-Key header.
  2. User spend-cap window.
  3. Atomic conditional hold at the locked, discounted rates, with the minimum hold per currency and the input allowance. With hold off for the org: a plain balance check and no holds row.

Conflicts return 409 or the stored response.

  • Spec: ../architecture/db-schema.md section 5 step 2; decisions log, Credit hold/settle mechanism, Hold/settle concurrency safety, Idempotency key design, Per-organization hold toggle
  • Done when: P3.09 passes, a repeated body within the window returns the stored response with no new hold, and over-cap or below-minimum requests roll back cleanly.

Who: Agent · Needs: P3.01, P3.04

  • Decrypt the key’s gateway credential (AES-256-GCM, with key id and gateway id as associated data).
  • Call the adapter and pass SSE through.
  • On client disconnect, keep Bifrost’s reported partial usage for settle and keep the Worker’s own count as a sanity check.
  • Spec: decisions log, Bifrost virtual key secret storage, Mid-stream client disconnect policy; ../architecture/db-schema.md 4.2
  • Done when: a ciphertext moved to another row fails to decrypt (test), and a disconnected stream settles on the reported partial usage.

Who: Agent · Needs: P3.05

Settle, in one transaction:

  • charge = tokens × locked rates × (1 − locked discount)
  • debit LEAST(charge, balance), recording any absorbed shortfall
  • reduce held
  • write the settlement and ledger rows
  • close the hold
  • adjust the spend window

Release on any failure or gateway rejection (including 429), with no retry.

  • Spec: ../architecture/db-schema.md section 5 step 3; decisions log, Charge basis vs cost basis, Hold close-out
  • Done when: after any mix of successes and failures, balance = SUM(ledger) and held = SUM(open holds), checked in tests.

Who: Agent · Needs: P3.06

Off the hot path, with ctx.waitUntil:

  • Queue: publish the v1 usage event, metadata only, carrying the gateway cost as COGS.
  • R2: write content when the workspace hasn’t opted out: one gzip object per request, keyed by the specified path layout. Buffer streams, mark a disconnect partial, truncate past the size cap. A failure never fails the request.
  • Idempotency: store the response and mark the row completed.

Decide in the PR: whether a failed queue publish retries or is only alerted on (backend-scope.md section 7; ledger correctness doesn’t depend on it).

  • Spec: decisions log, What is queued, Audit trail & prompt/response retention; ../architecture/db-schema.md 4.5
  • Done when:
    • events land on the staging queue
    • content objects appear in R2 for a logging workspace and not for an opted-out one
    • forcing an R2 failure still returns 200

Who: Agent · Needs: P3.07

  • POST /v1/chat/completions
  • GET /v1/models, scoped to the key
  • GET /v1/key: the key’s cap, usage and expiry, and the workspace balance, all in credits
  • POST /v1/messages (launch; accepts x-api-key as well as Bearer, errors in Anthropic’s format, api-errors.md section 7) and POST /v1/responses (launch), both verified end to end in P0.13

No CORS. Every response carries the request id.

  • Spec: ../product/api-surface.md sections 1, 2 and 4; decisions log, Credit balance & non-refundability (wording)
  • Done when: the OpenAI and Anthropic SDKs work against staging with only a base-URL change (for each endpoint that ships), and no response shows a currency symbol for a balance.

Who: Agent · Needs: P3.04, P3.06

N concurrent requests against one workspace, on real Postgres (the CI container, and once on staging), asserting no over-approval: total holds never exceed balance minus held, and the ledger balances afterwards. Include concurrent settles and releases, and the spend-cap window. Run it in the PR gates if it’s fast enough (cicd.md section 9), otherwise on every merge.

  • Spec: decisions log, Testing approach; ../architecture/cicd.md section 2
  • Done when: it passes at N = 100, and fails when the conditional WHERE is removed (to prove the test catches it).

Who: Agent · Needs: P3.08

Every conformance-suite item that applies to apps/api, plus the key-lifecycle tests in api-key-security.md section 9. Add:

P3.11 — M1: first charged request in staging

Section titled “P3.11 — M1: first charged request in staging”

Who: You + agent · Needs: P3.08, P3.09

  • Agent: add a canary script that sends one tiny request through the full path using the seeded internal workspace, then checks the settlement and ledger rows. It becomes the post-deploy smoke check and, later, the production canary.
  • You: confirm the charge against the published rate by hand once.
  • Spec: ../architecture/cicd.md section 6; ../architecture/observability.md section 6
  • Done when: the pipeline’s smoke step runs the canary on every staging deploy and it passes.

Who: Agent · Needs: P3.08

Public at launch (decided 2026-11-07). GET /v1/generation?id=<request id> returns the metadata of one past request, read from request_logs:

  • request id, time, endpoint, model, status code and error code
  • latency, whether it streamed, input and output tokens
  • the charge in credits (no currency symbol, no money wording)

Rules:

  • Scope: only requests from the calling key’s workspace; any other id returns 404, the same as an unknown id.
  • No content, ever. Prompts and responses are readable only by admins, and every read is audited (decisions log, Audit trail & prompt/response retention). There is no /generation/content.
  • Not fields we don’t log. request_logs has no attempt or fallback count, so the response has none until the schema adds one. Do not promise “attempts” in the docs.
  • Lag: rows are written after the response through the queue (P3.07), so a lookup straight after a request can return 404 briefly. Document it; clients retry.
  • Standard errors from api-errors.md; no CORS; every response carries the request id.
  • Spec: ../product/api-surface.md section 2; ../architecture/db-schema.md (request_logs); ../architecture/api-errors.md
  • Done when: a key reads its own request after a charged call in staging, gets 404 for another workspace’s id, no content or money symbol appears in any response, and it is in the curated OpenAPI spec and its contract test (D.06).