Skip to content

Phase 4 — Async Processing and Scheduled Jobs

Goal: everything that runs after the response or on a schedule:

  • the queue consumer that writes request logs and rollups
  • the single scheduled-job dispatcher
  • the checks that keep money honest: the ledger invariant, usage reconciliation and key reconciliation
  • the audit writer

Done when: in staging, every API call produces a request-log row and rollup increments, every job is registered and running on schedule with its run history recorded, and a deliberately unbilled call shows up as a discrepancy.

All of this lives in apps/dashboard-api, in its own modules (queue() and scheduled()), with no imports from dashboard route code.

ID Task Who Needs Status
P4.01 Usage queue consumer Agent P3.07, P1.10 merged — code #39; consumers on, with their dead-letter queues code #68
P4.02 Scheduled-job dispatcher Agent P1.04 merged — code #21
P4.03 Register the pricing jobs Agent P4.02, P2.05 merged — code #42 (both jobs registered, shipped disabled)
P4.04 Ledger invariant check and cleanup Agent P4.02, P3.06 merged — code #41; rollup check and batched purge code #98, code #104
P4.05 Usage reconciliation Agent P4.02, P3.06 merged — code #41; charging gives back the spend-window reservation code #99
P4.06 Key reconciliation Agent P4.02, P5.07 merged — code #41
P4.07 Audit writer and event catalogue Agent P1.05 merged — code #22, review fixes code #72; tenancy events written since code #75

Who: Agent · Needs: P3.07, P1.10

Consume usage-events through the logs store interface:

  • insert the request-log row with ON CONFLICT (created_at, id) DO NOTHING, where created_at comes from the event, never now()

  • update usage_hourly and content_storage_daily only when the insert happened, in the same transaction

  • set api_keys.last_used_at

  • accept the current and the previous event version

  • configure a dead-letter queue

  • Spec: decisions log, Log-write and audit rules, Queue consumer placement; ../architecture/db-schema.md 4.5 and 8.3 item 3; ../architecture/cicd.md section 3 rule 2

  • Done when: redelivering the same event twice changes nothing, rollups match the raw rows in a test, and a malformed event lands in the dead-letter queue.

Who: Agent · Needs: P1.04

One Cron Trigger every minute:

  • select jobs that are due and enabled

  • claim each with a lease (UPDATE … RETURNING) and run it

  • record a job_runs row, then set the next run

  • reclaim an expired lease as a failed run

  • “run now” uses the same claim, so it can’t overlap a scheduled run

  • Spec: ../architecture/db-schema.md 8.3 item 8; decisions log, Scheduled jobs, Scheduled jobs admin page

  • Done when: two dispatcher invocations at once never run the same job twice (test), and a job that throws is recorded as failed and runs again on schedule.

Who: Agent · Needs: P4.02, P2.05

Register the FX, model catalog sync and daily rates jobs from phase 2, with schedules and per-job config in scheduled_jobs.

P4.04 — Ledger invariant check and cleanup

Section titled “P4.04 — Ledger invariant check and cleanup”

Who: Agent · Needs: P4.02, P3.06

  • Daily, per workspace: balance = SUM(ledger_entries.amount) and held = SUM(open holds), alerting on any difference.

  • In the same job: delete expired idempotency keys, sessions and verifications.

  • Spec: ../architecture/db-schema.md 8.3 items 1 and 6; decisions log, Ledger invariant check

  • Done when: a hold left open on purpose is reported, and expired rows are removed.

Who: Agent · Needs: P4.02, P3.06

Hourly, per enabled gateway:

  • page its usage logs through the adapter, from the watermark to now minus 15 minutes

  • match each log id (our request id) against settlements

  • a missing settlement becomes not_charged; different token counts become token_mismatch

  • advance the watermark only after the window is fully read

  • never charge automatically

  • Spec: ../architecture/db-schema.md 8.3 item 7 and usage_discrepancies; decisions log, Usage reconciliation (Bifrost vs ledger)

  • Done when: a call made straight to staging Bifrost with a test virtual key appears as not_charged.

Who: Agent · Needs: P4.02, P5.07

Compare every key’s gateway credential with what the gateway holds. Write gateway_discrepancies rows:

  • stuck pending
  • missing on either side
  • revoked but still live
  • config drift (models, limits, budget)

Auto-clear a row when the mismatch disappears. The same job backfills credentials if a gateway is added.

  • Spec: ../architecture/db-schema.md 4.2; decisions log, Key ↔ Bifrost sync and failure path, Gateway abstraction
  • Done when: each mismatch type, created by hand in staging, is detected and then cleared after a resync.

P4.07 — Audit writer and event catalogue

Section titled “P4.07 — Audit writer and event catalogue”

Who: Agent · Needs: P1.05

A single function that writes privileged audit events directly (not through the queue), in the same request: actor, impersonated_by, target, action, reason, IP. Define the action catalogue. It must include:

  • content.read, written before content is returned

  • rate_limit.changed, job.updated, job.run_manually

  • the money, key and member events listed in the build inventory

  • Spec: ../architecture/db-schema.md 8.3 items 4 and 5; ../product/build-inventory.md Audit events, Admin audit

  • Done when: a failed audit write blocks the action it audits (test for content.read).