Phase 4 operator platform

Separate advisory estimates from reconciled actuals, enforce atomic durable hard caps, and make capacity observable.

Milestone 4.D.6 — Usage, Cost Governance & Capacity

Status: Planned and blocked on the actuals/reconciliation and enforcement contracts. Estimates may be shown as advisory only; hard-cap admission remains disabled until atomicity and observability evidence pass. The repository baseline is 8f8e130 plus pre-existing review documents; no implementation evidence is claimed.

Outcome: Operators can distinguish estimates from reconciled actual cost, identify the named source and watermark behind every number, enforce durable tenant-scoped budgets without concurrent over-admission or double charging, and observe the capacity signals needed to operate those controls.

Track: Track D — Operator Capability Slices

Dependencies: 4.D.1; 4.D.2 recommended for pivots; 4.P.2 durable usage facts; 4.P.4 ledger/query and watermark contract; 4.P.5 resource controls; rate-card ownership; capacity metric and alert ownership.

Estimated effort: 3 weeks after prerequisites are accepted.

Purpose

Make spend and capacity decisions explainable, replayable, and tenant-safe without turning an incomplete projection or ephemeral counter into an authorization decision.

Scope

  • Usage ledger UI with estimate/actual distinction, versioned rate cards, provider/model/fallback attribution, rollups, budget scopes/periods, and completeness labels.
  • Named actuals source contract: fact identity, rate-card/policy version, source watermark, generated time, reconciliation state, late/duplicate/unknown counts, and correction policy.
  • Durable budget/enforcement decisions and an atomic hard-cap admission path with idempotency, retry safety, and concurrent over-admission protection.
  • Reconciliation jobs/reports between raw facts, aggregates, hot counters, enforcement decisions, and exported cost evidence.
  • Queue depth, worker concurrency, provider capacity, admission latency, denials, lag, saturation, and resource/cost ceiling observability with tenant-safe dimensions.

Non-goals

  • No presentation of estimated spend as billing actuals, no silent filling of missing facts, and no claim of reconciliation from a chart or fixture.
  • No hard-cap authorization based solely on Redis, client state, eventual chart rollups, or a non-atomic read-then-write sequence.
  • No provider invoice settlement, tax/accounting ledger, or pricing change without an owned rate-card/version contract.
  • No unbounded high-cardinality dashboard, raw prompt/content exposure, or cross-tenant aggregate query.

Hard prerequisites

  1. Usage fact identity/cardinality and retry/duplicate semantics are canonical; all queries and writes require org_id.
  2. A named authoritative actuals source and watermark contract define what is complete, delayed, sampled, corrected, or unavailable.
  3. Rate-card and policy versions are immutable/replayable, with ownership and effective-time semantics.
  4. Hard-cap admission is a single atomic durable decision with reservation/commit/release or equivalent semantics, and its failure mode is fail-closed or explicitly approved.
  5. Capacity metric definitions, cardinality bounds, SLOs, alert owners, and rollback-to-alert-only procedure are approved.

Deliverables

  1. Separate estimated and actual cost DTOs/UI states with source, watermark, reconciliation, completeness, freshness, and late/duplicate metadata.
  2. Durable usage ledger, rate-card replay, idempotent ingestion, correction/reconciliation report, and exportable evidence.
  3. Atomic hard-cap admission command, durable enforcement decision, retry/idempotency behavior, and concurrent admission test harness.
  4. Capacity/queue/resource observability dashboards and alerts with bounded labels, tenant scope, lag, saturation, and denial reason.
  5. Runbook for false denies, source lag, reconciliation mismatch, capacity exhaustion, rollback-to-alert-only, and recovery.

Acceptance evidence (machine-verifiable)

  • Contract tests prove every actual response names its source, watermark, rate-card/policy version, reconciliation status, completeness, generated time, and late/duplicate counts; estimates cannot populate actual fields.
  • Raw-fact-to-aggregate reconciliation on a golden period reports totals, missing/late/duplicate/corrected facts, bounded tolerance, and a reproducible watermark; reruns and retries do not double charge.
  • Rate-card replay reproduces expected costs for provider/model/fallback cases, including version changes and corrections.
  • Concurrent admission tests prove the hard cap cannot over-admit across requests, workers, retries, or failover; every allow/deny has one durable enforcement decision and reason.
  • Cross-tenant tests prove usage, budget, capacity, and enforcement queries cannot cross org_id; missing organization context fails closed.
  • Capacity tests and dashboards expose queue depth, worker concurrency, admission latency, source lag, saturation, denials, and resource ceilings with bounded cardinality; rollback-to-alert-only and recovery evidence is attached.

Completion rule

Complete only when estimates and actuals are distinct in contract, storage, UI, and tests; the named-source/watermark/reconciliation contract is implemented; atomic hard-cap admission passes concurrent and failure tests; and capacity observability plus rollback evidence is operational. Until then, governance remains alert-only/advisory and no hard-cap completion claim is valid.

PR sequence

  1. Decide usage-fact identity, authoritative actuals source, watermark/completeness states, rate-card ownership, and correction semantics.
  2. Implement separate estimate/actual DTOs, durable ledger, versioned rate cards, idempotent ingestion, and reconciliation report.
  3. Implement atomic durable admission/enforcement decisions with concurrent, retry, failover, and cross-tenant tests.
  4. Add bounded capacity/queue/resource metrics, dashboards, alerts, and source-lag observability.
  5. Run false-deny, mismatch, exhaustion, rollback-to-alert-only, and recovery drills before enabling hard caps.

Open questions

  • Which system is the named authoritative source for actual usage/cost, and what watermark proves completeness?
  • How are late, duplicate, corrected, disputed, or provider-invoice facts represented and reconciled?
  • Which atomic admission/reservation primitive and failure mode are approved for each budget scope?
  • What are the tenant, provider, worker, and queue capacity SLOs and acceptable cardinality bounds?
  • Who may move a tenant from hard cap to alert-only during recovery, and what evidence permits re-enablement?

Rollout and rollback

Trigger: false deny, over-admission, chart/ledger mismatch, source-watermark regression, tenant-limit leak, or capacity exhaustion. Reversal: stop new hard governance denies from the affected path and move affected tenants to alert-only/soft enforcement while preserving durable decisions. Verify: reconcile decisions against facts and watermarks, retain informational charts with honest completeness labels, and re-enable hard caps only after concurrent and recovery evidence passes.

Findings addressed

F4-005.

Exit checklist

  • Estimates and actuals are visibly and contractually separate.
  • Named source, watermark, reconciliation, completeness, and late/duplicate evidence is attached.
  • Atomic hard-cap admission and no-double-charge/concurrency evidence is attached.
  • Capacity observability and rollback-to-alert-only recovery are demonstrated.
Edit on GitHub

Last updated on