ADR-0061: Retrieval-quality benchmark methodologyCopy Anchor Link

• Status: Accepted
• Date: 2026-08-31
• Authors: IBEX Harness team
• Milestone: 3.E.3 Retrieval-quality benchmark sign-off

ContextCopy Anchor Link

Phase 3 Track B ships HNSW recall/latency methodology in ADR-0053 §10 (strict corpus reset, production cells, ef_search=40). Milestone 3.E.3 adds permanent CI regression gates for:

1. HNSW recall/latency — enforce persisted status on published runs
2. Ranking quality — gold-set precision/recall/MRR vs committed baseline
3. Write pipeline — wall-clock MemoryWriteOrchestrator.create() p95 vs SLA + baseline

Proxy load benchmarks remain ADR-0034 / ADR-0024. This ADR covers the memory substrate suites only.

Options consideredCopy Anchor Link

1) Gate enforcement locationCopy Anchor Link

1. Schema-validate published JSON only — gates could pass with status: fail in history.
2. Dedicated enforce steps that exit non-zero before merge (chosen).

Decision: (2). HNSW: check_hnsw_gate_status.py reads top-level runs[].status from hnsw-benchmark-data.json. Ranking/write: regression_gate.py / write_pipeline/regression_gate.py return exit code 1 on policy failure.

2) Ranking gold set designCopy Anchor Link

1. Live embedder + human labels at CI time — non-deterministic, costly.
2. Committed gold_set_v1.json with deterministic zero_embedding(hotspot) (chosen).

Decision: (2). Fifty memories, twenty queries, single-labeler v1 disclaimer (ranking_quality/README.md). validate_gold_set.py runs in CI before the bench; composite-ground-truth order is validated against production composite_score().

3) Regression tolerance policyCopy Anchor Link

1. Hard-fail only in bench (100% metrics) — regression gate never exercised on real variance.
2. Committed baseline.json + max_regression_pct policy with synthetic gate tests (chosen).

Decision: (2). Ranking: max_regression_pct: 5.0 on precision@5, recall@10, MRR, top_category_accuracy. Write: max_latency_ms_p95: 200 absolute SLA plus max_regression_pct: 20.0 on p95 latency. Synthetic failure paths tested in test_regression_gate.py and write_pipeline/test_regression_gate.py.

4) Published history deliveryCopy Anchor Link

1. Per-suite data PRs on contributor branches.
2. Single bot PR on chore/bench-data-publish upserting all suite JSON files (chosen).

Decision: (2) per ADR-0024. Bot posts one sticky PR comment with per-suite matrix + deep dives.

DecisionCopy Anchor Link

HNSW suite (benchmarks/memory/)Copy Anchor Link

Enforcement: check_hnsw_gate_status.py after publish build; fails CI when status != pass.

Ranking-quality suite (benchmarks/memory/ranking_quality/)Copy Anchor Link

Bench hard-fails when any query misses perfect metrics; regression gate compares latest run to baseline for hardware variance tolerance.

Write-pipeline suite (benchmarks/memory/write_pipeline/)Copy Anchor Link

CI wiringCopy Anchor Link

• Workflow: .github/workflows/memory-benchmark.yml
• Jobs: collect-hnsw-benchmarks, collect-ranking-quality, collect-write-pipeline-bench
• Publish on gate failure: build steps use if: always() so failed runs still record status: fail in published JSON
• Bot: ranking/write/HNSW PR comments + combined data PR (#660 pattern)

ConsequencesCopy Anchor Link

Positive

• Three memory suites share one enforcement pattern: run bench → regression gate → publish → validate.
• Synthetic gate tests prove failure paths without relying on live-data luck.
• Dashboard history JSON is bot-published on main only — PR branches use artifacts + comments.

Negative

• Gold set v1 is single-labeler, small-N — not a Phase 5 inter-rater gold standard.
• Ranking bench always expects 100% metrics; the 5% regression band only helps CI hardware drift, not partial quality regressions caught solely by the bench script.
• HNSW 1M cells run on full profile only; smoke/fast PR CI defers 1M (documented in bot comments as "1M deferred").
• Stub published JSON on feature branches can conflict with bot chore/bench-data-publish until merged — resolve by keeping bot branch rebased on main.

ReferencesCopy Anchor Link

• 3.E.3 milestone
• ADR-0053 §10 — HNSW bench methodology (Track B)
• ADR-0024 — bot publish model
• PR #654 — 3.E.3 implementation
• benchmarks/memory/README.md — suite index