Convert recall/latency and ranking-quality benchmarks into permanent CI regression gates with committed baselines, gold-set labeled queries, and pass/fail thresholds.
Milestone 3.E.3 — Retrieval-Quality Benchmark Sign-Off
Status: Complete
Goal: Track E — Exit Gate
Phase: 3 — Core Memory Substrate
Estimated effort: 2 days
ADR: ADR-0061
What Gets Committed
benchmarks/memory/
hnsw_bench.py, hnsw_run.py, publish_cells.py, build_published_data.py # existing HNSW stack
scripts/check_hnsw_gate_status.py # CI enforces persisted gate status
ranking_quality/
gold_set_v1.json # ~50 memories, ~21 queries (v1 single-labeler)
seed.py, metrics.py, bench_ranking_quality.py, regression_gate.py, baseline.json
README.md # v1 disclaimer
write_pipeline/
bench_write_pipeline.py, regression_gate.py, baseline.jsonHNSW measurement code and SLA constants are unchanged; the enforcement gap was closed by
check_hnsw_gate_status.py reading runs[].status from published JSON.
CI Wiring
.github/workflows/memory-benchmark.yml:
| Job | Purpose |
|---|---|
collect-hnsw-benchmarks | HNSW recall/latency + gate status enforcement + schema validate |
collect-ranking-quality | Gold-set ranking bench + regression gate |
collect-write-pipeline-bench | Write orchestrator wall-clock p95 + regression gate |
Profiles: smoke (PR), fast (main push), full (weekly cron) for HNSW sizes.
Sign-Off Gate
| Metric | Threshold | Enforced by |
|---|---|---|
| HNSW recall@10 (published cells) | ≥ 98% | check_hnsw_gate_status.py on gate_summary |
| Vector search p99 at 1M (when cell present) | < 100ms | same |
Write pipeline p95 (orch.create() wall-clock) | < 200ms | write_pipeline/regression_gate.py SLA |
| Ranking precision@5 / recall@10 / MRR | ≥ baseline − 5% | ranking_quality/regression_gate.py |
| Gold set | v1 single-labeler | See ranking_quality README |
Category-decay regression queries in gold set v1: q_pref_theme_decay, q_workflow_deploy_decay, q_confidence_tie_break (distinct from 3.D.2 dark-mode pair).
Orientation (indicative)
Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.
During implementation, expect to:
- open the live tree and follow existing patterns before inventing new ones
- research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
- advance the design beyond the sketch where measurement or code reality says so
- land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it
Prefer outcomes over matching any particular file tree or command sequence.
Areas that may be involved (situational — not a checklist):
- Evaluation / benchmarks
- Memory service / repositories
Success signals
Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.
- Benchmark scripts, gold set, baseline, and regression gate committed
- CI job
memory-benchmark.ymlrunning on weekly schedule + PR overlay (HNSW + ranking + write) - Sign-off thresholds wired to regression gates (baselines pinned; CI may refine on first run)
- Repo guards / CI checks still pass
Prerequisites
- All Track A–D milestones
- Milestone 3.E.1 (security integration suite)
Last updated on