ADR-0067: Context capability catalog generate-and-diff for token budgetsCopy Anchor Link

• Status: Accepted
• Date: 2026-09-04
• Authors: IBEX Harness team
• Milestone: 3.5.C.1 Token budget calculator
• Depends on: ADR-0041, ADR-0043
• Follow-up: #690 Exact HF/Qwen tokenizer-family counting

ContextCopy Anchor Link

Milestone 3.5.C.1 needs model-aware token budgets inside the future Python context-assembly package. Discovery confirmed:

• services/context/ did not exist yet; Track C defers the gRPC wrapper to 3.5.C.6.
• Go packages/provider / packages/tokenizer are proxy-in-process only — no RPC/HTTP surface for capability lookup or counting (ADR-0041, ADR-0043).
• Python services have no tiktoken / HF tokenizers dependencies today.
• The only Go→Python network precedent is Auth gRPC via packages/authclient — not applicable until a tokenizer/capability service exists.

A live TokenizerClient / CapabilityRegistry in the C.1 sketch would invent a network boundary that 3.5.C.6 must own.

DecisionCopy Anchor Link

1) Generate-and-diff JSON export (not a network call)Copy Anchor Link

• Maintainer tool: go run ./packages/provider/scripts/export_capabilities
• Committed artifact: services/context/app/data/model_capabilities.v1.json (package data for installed wheels)
• Contents: sorted BuiltInCapabilityCatalog rows + tokenizer_families estimate policy (estimate_kind, safety_buffer_fraction)
• CI job capability-catalog-freshness runs -check and fails when the committed JSON drifts from the Go catalog

2) Python BudgetCalculator loads the artifact locallyCopy Anchor Link

• Package layout under services/context/app/ (library only — no gRPC server in C.1)
• Constructor takes an optional in-memory catalog; default loads the committed JSON
• Unknown model IDs raise explicitly (no silent fallback model)

3) Labeled character/rune estimates (not exact tokenization)Copy Anchor Link

TokenBudget.estimate_kind always records which formula ran. Exact OpenAI/HF counting is out of scope — tracked in #690.

4) Explicit non-goals for C.1Copy Anchor Link

• No tokenizer/capability gRPC or HTTP service
• No packages/contextclient / ContextAssemblyService implementation (3.5.C.6)
• No hand-maintained duplicate of the Go catalog in Python source

ConsequencesCopy Anchor Link

Positive:

• Python budgets stay aligned with the Go catalog via CI fail-closed drift detection
• C.1 stays within Track C sequencing (assembly math before the gRPC façade)
• Estimate error is visible (estimate_kind + family safety buffers)

Negative / follow-ups:

• Estimates are approximate; do not treat them as tiktoken-ground-truth
• Operator IBEX_MODEL_CAPABILITY_OVERLAYS on the proxy are not mirrored into the JSON export (built-in catalog only) until a later milestone extends the export
• #690 for exact HF/Qwen (and tighter OpenAI) counting