Billing and budgets
Billing is split across configuration, admission, usage facts, and reconciliation. Each has a different authority; combining them into one “spend” number without naming the source is how billing bugs become incidents.
Components and authority
| Component | Store | Responsibility | Status |
|---|---|---|---|
| Rate cards | ibex_billing.rate_cards, rate_card_versions | Org price catalog; published versions are immutable | Implemented locally |
| Budget periods | ibex_billing.budget_periods | Cap, cached spend, and alert_only or hard_cap mode | Implemented locally |
| Enforcement decisions | ibex_billing.enforcement_decisions | Allow/deny/unavailable audit trail | Implemented locally |
| Admission holds | ibex_core.budget_reservations | Durable hard-cap capacity while a request is live | Implemented locally |
| Usage facts | ClickHouse ibex.usage_facts | Request/token/cost facts; no TTL | Implemented locally |
| Reconciliation | Worker rollup + cache invalidation | Refresh spent_cents_cached from usage facts | Implemented locally |
| Invoice actuals | — | Provider invoice reconciliation | Deferred, no invoice source (#859) |
The billing tables are intentionally protected from organization-deletion cascade (the billing carve-out in migration 000038). Billing records can outlive the operational organization data; do not describe org deletion as erasing every billing row.
Rate cards
A rate card belongs to an organization and has draft, published, or archived status. Publishing creates a new immutable rate_card_versions row with prices as JSONB entries containing provider, model pattern, and input/output cents per 1k tokens. The version number is the next maximum for that card.
A published version is not a guarantee that the proxy charged it. Verify which version and model pattern were used when reconciling a disputed amount.
Hard-cap admission
The proxy consults the budget cache first. Organizations without an active hard_cap period skip the ledger entirely (no_cap). alert_only periods do not reserve capacity.
For a hard-cap organization, a claim:
- Opens a transaction and resolves the active period.
- Takes a per-org advisory lock.
- Resolves the row for
(org_id, agent_id, reservation_key)underFOR UPDATE. - Compares
spent_cents_cached + SUM(live claimed holds)with the cap. - Persists or reclaims a
claimedhold with a lease.
Reservation keys and request hashes must be non-empty; keys are capped at 256 characters, and negative estimates are rejected before querying. HashRequest is a SHA-256 hex fingerprint of a canonical request payload.
| Outcome | Meaning | Client behavior |
|---|---|---|
owned | Fresh hold owned by this request | Continue to provider |
in_progress | Same key is held by another owner | Retry after RetryAfter; do not admit twice |
conflict | Same key, different request hash | Reject the request |
exceeded | Hold would cross the active cap | 402 BUDGET_EXCEEDED |
no_cap | No active hard-cap period | Continue without a hold |
| Ledger/query failure | Authority unavailable | 503 BUDGET_UNAVAILABLE; fail closed |
The default lease is 15 minutes, sized to cover long streaming calls. A worker reaper expires stale claims only after the lease plus a 1-hour grace, in batches of 500, so a slow but live owner can still settle.
Finalize, release, and overshoot
| Settling action | Effect |
|---|---|
Finalize(complete) | Provider usage supplied the amount |
Finalize(estimated) | Provider supplied no usage; admission estimate is the best available amount |
Release | Failed/canceled request; no settled amount |
Finalize and release are idempotent for an identical repeat and return ErrClaimClosed otherwise. A reclaimed reservation cannot be settled by the old owner. Never record a provider response without usage as complete.
spent_cents_cached updates only on the next rollup. Between settlement and rollup there is a bounded overshoot window. This is a documented limitation, not exact enforcement.Reconciliation
budget_spent_rollup lists active periods and sums estimated_cost_cents from ibex.usage_facts in ClickHouse for each org and period. The query must include org_id explicitly because ClickHouse has no RLS. The worker then updates spent_cents_cached and publishes budget_updates:{org_id} so the proxy refreshes its cache.
reconcile_usage_actuals is currently deferred because there is no provider invoice source. A cached spend number is therefore a rollup of IBEX usage facts, not a provider invoice.
Known gaps
- Extraction usage is not exported to
ibex.usage_facts, so completed extraction spend is missing from the rollup. - Chat and worker enqueue are admission-time dedupe, not exactly-once provider or Celery execution.
- Crash/replay, cross-language canonicalization, broader operation rollout, and independent review remain open.
- ClickHouse is a projection for usage facts but not a substitute for the reservation ledger.
- GAP-006 remains open; billing is not invoice-grade or hosted-reconciled.
Operator checks
go test -race ./packages/budgetledger/...
go test ./packages/billing/...
go test ./packages/usagequery/...
Inspect ibex_core.budget_reservations for claimed, finalized, released, and expired rows. Check ibex_billing.enforcement_decisions for deny/unavailable reasons. A stale claim is not automatically safe to finalize; only the lease expiry plus grace makes it reclaimable, and a new owner's version fences the old one.
Related
- Failure modes and recovery — ledger outage behavior
- Evidence and relay — separate evidence authority
- Data lifecycle and deletion — billing carve-out and legal holds
- Management API — rate-card and budget routes
- Observability — metrics for denials and rollups