Phase 4 — Async Processing and Scheduled Jobs
Goal: everything that runs after the response or on a schedule:
- the queue consumer that writes request logs and rollups
- the single scheduled-job dispatcher
- the checks that keep money honest: the ledger invariant, usage reconciliation and key reconciliation
- the audit writer
Done when: in staging, every API call produces a request-log row and rollup increments, every job is registered and running on schedule with its run history recorded, and a deliberately unbilled call shows up as a discrepancy.
All of this lives in apps/dashboard-api, in its own modules (queue() and scheduled()), with no imports from dashboard route code.
| ID | Task | Who | Needs | Status |
|---|---|---|---|---|
| P4.01 | Usage queue consumer | Agent | P3.07, P1.10 | merged — code #39; consumers on, with their dead-letter queues code #68 |
| P4.02 | Scheduled-job dispatcher | Agent | P1.04 | merged — code #21 |
| P4.03 | Register the pricing jobs | Agent | P4.02, P2.05 | merged — code #42 (both jobs registered, shipped disabled) |
| P4.04 | Ledger invariant check and cleanup | Agent | P4.02, P3.06 | merged — code #41; rollup check and batched purge code #98, code #104 |
| P4.05 | Usage reconciliation | Agent | P4.02, P3.06 | merged — code #41; charging gives back the spend-window reservation code #99 |
| P4.06 | Key reconciliation | Agent | P4.02, P5.07 | merged — code #41 |
| P4.07 | Audit writer and event catalogue | Agent | P1.05 | merged — code #22, review fixes code #72; tenancy events written since code #75 |
P4.01 — Usage queue consumer
Section titled “P4.01 — Usage queue consumer”Who: Agent · Needs: P3.07, P1.10
Consume usage-events through the logs store interface:
-
insert the request-log row with
ON CONFLICT (created_at, id) DO NOTHING, wherecreated_atcomes from the event, nevernow() -
update
usage_hourlyandcontent_storage_dailyonly when the insert happened, in the same transaction -
set
api_keys.last_used_at -
accept the current and the previous event version
-
configure a dead-letter queue
-
Spec: decisions log, Log-write and audit rules, Queue consumer placement;
../architecture/db-schema.md4.5 and 8.3 item 3;../architecture/cicd.mdsection 3 rule 2 -
Done when: redelivering the same event twice changes nothing, rollups match the raw rows in a test, and a malformed event lands in the dead-letter queue.
P4.02 — Scheduled-job dispatcher
Section titled “P4.02 — Scheduled-job dispatcher”Who: Agent · Needs: P1.04
One Cron Trigger every minute:
-
select jobs that are due and enabled
-
claim each with a lease (
UPDATE … RETURNING) and run it -
record a
job_runsrow, then set the next run -
reclaim an expired lease as a failed run
-
“run now” uses the same claim, so it can’t overlap a scheduled run
-
Spec:
../architecture/db-schema.md8.3 item 8; decisions log, Scheduled jobs, Scheduled jobs admin page -
Done when: two dispatcher invocations at once never run the same job twice (test), and a job that throws is recorded as failed and runs again on schedule.
P4.03 — Register the pricing jobs
Section titled “P4.03 — Register the pricing jobs”Who: Agent · Needs: P4.02, P2.05
Register the FX, model catalog sync and daily rates jobs from phase 2, with schedules and per-job config in scheduled_jobs.
- Spec:
../architecture/backend-scope.mdsection 5 - Done when: in staging, rates publish daily without manual runs, and the history is in
job_runs.
P4.04 — Ledger invariant check and cleanup
Section titled “P4.04 — Ledger invariant check and cleanup”Who: Agent · Needs: P4.02, P3.06
-
Daily, per workspace:
balance = SUM(ledger_entries.amount)andheld = SUM(open holds), alerting on any difference. -
In the same job: delete expired idempotency keys, sessions and verifications.
-
Spec:
../architecture/db-schema.md8.3 items 1 and 6; decisions log, Ledger invariant check -
Done when: a hold left open on purpose is reported, and expired rows are removed.
P4.05 — Usage reconciliation
Section titled “P4.05 — Usage reconciliation”Who: Agent · Needs: P4.02, P3.06
Hourly, per enabled gateway:
-
page its usage logs through the adapter, from the watermark to now minus 15 minutes
-
match each log id (our request id) against
settlements -
a missing settlement becomes
not_charged; different token counts becometoken_mismatch -
advance the watermark only after the window is fully read
-
never charge automatically
-
Spec:
../architecture/db-schema.md8.3 item 7 andusage_discrepancies; decisions log, Usage reconciliation (Bifrost vs ledger) -
Done when: a call made straight to staging Bifrost with a test virtual key appears as
not_charged.
P4.06 — Key reconciliation
Section titled “P4.06 — Key reconciliation”Who: Agent · Needs: P4.02, P5.07
Compare every key’s gateway credential with what the gateway holds. Write gateway_discrepancies rows:
- stuck
pending - missing on either side
- revoked but still live
- config drift (models, limits, budget)
Auto-clear a row when the mismatch disappears. The same job backfills credentials if a gateway is added.
- Spec:
../architecture/db-schema.md4.2; decisions log, Key ↔ Bifrost sync and failure path, Gateway abstraction - Done when: each mismatch type, created by hand in staging, is detected and then cleared after a resync.
P4.07 — Audit writer and event catalogue
Section titled “P4.07 — Audit writer and event catalogue”Who: Agent · Needs: P1.05
A single function that writes privileged audit events directly (not through the queue), in the same request: actor, impersonated_by, target, action, reason, IP. Define the action catalogue. It must include:
-
content.read, written before content is returned -
rate_limit.changed,job.updated,job.run_manually -
the money, key and member events listed in the build inventory
-
Spec:
../architecture/db-schema.md8.3 items 4 and 5;../product/build-inventory.mdAudit events, Admin audit -
Done when: a failed audit write blocks the action it audits (test for
content.read).