Separate advisory estimates from reconciled actuals, enforce atomic durable hard caps, and make capacity observable.
Milestone 4.D.6 — Usage, Cost Governance & Capacity
Status: Planned and blocked on the actuals/reconciliation and enforcement contracts. Estimates may be shown as advisory only; hard-cap admission remains disabled until atomicity and observability evidence pass. The repository baseline is 8f8e130 plus pre-existing review documents; no implementation evidence is claimed.
Outcome: Operators can distinguish estimates from reconciled actual cost, identify the named source and watermark behind every number, enforce durable tenant-scoped budgets without concurrent over-admission or double charging, and observe the capacity signals needed to operate those controls.
Track: Track D — Operator Capability Slices
Dependencies: 4.D.1; 4.D.2 recommended for pivots; 4.P.2 durable usage facts; 4.P.4 ledger/query and watermark contract; 4.P.5 resource controls; rate-card ownership; capacity metric and alert ownership.
Estimated effort: 3 weeks after prerequisites are accepted.
Purpose
Make spend and capacity decisions explainable, replayable, and tenant-safe without turning an incomplete projection or ephemeral counter into an authorization decision.
Scope
- Usage ledger UI with estimate/actual distinction, versioned rate cards, provider/model/fallback attribution, rollups, budget scopes/periods, and completeness labels.
- Named actuals source contract: fact identity, rate-card/policy version, source watermark, generated time, reconciliation state, late/duplicate/unknown counts, and correction policy.
- Durable budget/enforcement decisions and an atomic hard-cap admission path with idempotency, retry safety, and concurrent over-admission protection.
- Reconciliation jobs/reports between raw facts, aggregates, hot counters, enforcement decisions, and exported cost evidence.
- Queue depth, worker concurrency, provider capacity, admission latency, denials, lag, saturation, and resource/cost ceiling observability with tenant-safe dimensions.
Non-goals
- No presentation of estimated spend as billing actuals, no silent filling of missing facts, and no claim of reconciliation from a chart or fixture.
- No hard-cap authorization based solely on Redis, client state, eventual chart rollups, or a non-atomic read-then-write sequence.
- No provider invoice settlement, tax/accounting ledger, or pricing change without an owned rate-card/version contract.
- No unbounded high-cardinality dashboard, raw prompt/content exposure, or cross-tenant aggregate query.
Hard prerequisites
- Usage fact identity/cardinality and retry/duplicate semantics are canonical; all queries and writes require
org_id. - A named authoritative actuals source and watermark contract define what is complete, delayed, sampled, corrected, or unavailable.
- Rate-card and policy versions are immutable/replayable, with ownership and effective-time semantics.
- Hard-cap admission is a single atomic durable decision with reservation/commit/release or equivalent semantics, and its failure mode is fail-closed or explicitly approved.
- Capacity metric definitions, cardinality bounds, SLOs, alert owners, and rollback-to-alert-only procedure are approved.
Deliverables
- Separate estimated and actual cost DTOs/UI states with source, watermark, reconciliation, completeness, freshness, and late/duplicate metadata.
- Durable usage ledger, rate-card replay, idempotent ingestion, correction/reconciliation report, and exportable evidence.
- Atomic hard-cap admission command, durable enforcement decision, retry/idempotency behavior, and concurrent admission test harness.
- Capacity/queue/resource observability dashboards and alerts with bounded labels, tenant scope, lag, saturation, and denial reason.
- Runbook for false denies, source lag, reconciliation mismatch, capacity exhaustion, rollback-to-alert-only, and recovery.
Acceptance evidence (machine-verifiable)
- Contract tests prove every actual response names its source, watermark, rate-card/policy version, reconciliation status, completeness, generated time, and late/duplicate counts; estimates cannot populate actual fields.
- Raw-fact-to-aggregate reconciliation on a golden period reports totals, missing/late/duplicate/corrected facts, bounded tolerance, and a reproducible watermark; reruns and retries do not double charge.
- Rate-card replay reproduces expected costs for provider/model/fallback cases, including version changes and corrections.
- Concurrent admission tests prove the hard cap cannot over-admit across requests, workers, retries, or failover; every allow/deny has one durable enforcement decision and reason.
- Cross-tenant tests prove usage, budget, capacity, and enforcement queries cannot cross
org_id; missing organization context fails closed. - Capacity tests and dashboards expose queue depth, worker concurrency, admission latency, source lag, saturation, denials, and resource ceilings with bounded cardinality; rollback-to-alert-only and recovery evidence is attached.
Completion rule
Complete only when estimates and actuals are distinct in contract, storage, UI, and tests; the named-source/watermark/reconciliation contract is implemented; atomic hard-cap admission passes concurrent and failure tests; and capacity observability plus rollback evidence is operational. Until then, governance remains alert-only/advisory and no hard-cap completion claim is valid.
PR sequence
- Decide usage-fact identity, authoritative actuals source, watermark/completeness states, rate-card ownership, and correction semantics.
- Implement separate estimate/actual DTOs, durable ledger, versioned rate cards, idempotent ingestion, and reconciliation report.
- Implement atomic durable admission/enforcement decisions with concurrent, retry, failover, and cross-tenant tests.
- Add bounded capacity/queue/resource metrics, dashboards, alerts, and source-lag observability.
- Run false-deny, mismatch, exhaustion, rollback-to-alert-only, and recovery drills before enabling hard caps.
Open questions
- Which system is the named authoritative source for actual usage/cost, and what watermark proves completeness?
- How are late, duplicate, corrected, disputed, or provider-invoice facts represented and reconciled?
- Which atomic admission/reservation primitive and failure mode are approved for each budget scope?
- What are the tenant, provider, worker, and queue capacity SLOs and acceptable cardinality bounds?
- Who may move a tenant from hard cap to alert-only during recovery, and what evidence permits re-enablement?
Rollout and rollback
Trigger: false deny, over-admission, chart/ledger mismatch, source-watermark regression, tenant-limit leak, or capacity exhaustion. Reversal: stop new hard governance denies from the affected path and move affected tenants to alert-only/soft enforcement while preserving durable decisions. Verify: reconcile decisions against facts and watermarks, retain informational charts with honest completeness labels, and re-enable hard caps only after concurrent and recovery evidence passes.
Findings addressed
F4-005.
Exit checklist
- Estimates and actuals are visibly and contractually separate.
- Named source, watermark, reconciliation, completeness, and late/duplicate evidence is attached.
- Atomic hard-cap admission and no-double-charge/concurrency evidence is attached.
- Capacity observability and rollback-to-alert-only recovery are demonstrated.
Last updated on