Phase 3.5 extraction & assembly
Gold-set eval harness measuring extraction precision/recall per category, category-assignment accuracy, and temporal-field accuracy — CI-gated OpenAI cassette eval with manual vLLM side-by-side baselines.
Milestone 3.5.B.4 — Extraction Quality Evaluation Harness
Status: Completed
Goal: Track B — Extraction Pipeline
Phase: 3.5 — Extraction & Context Assembly
Estimated effort: 3 days
Track: Track B — Extraction Pipeline
Depends on: 3.5.B.2
ADR: ADR-0066
Why This Milestone Exists
Parseability checks do not measure extraction correctness. This milestone is the extraction-side counterpart to Phase 3 retrieval-quality gating (ADR-0061).
Non-Goals
- Real-time extraction quality monitoring in the operator dashboard (Phase 4)
- A/B testing infrastructure for prompt variants
- GPU CI runners (still none — mirrors 3.5.B.2)
Design (as implemented)
Layout: services/worker/eval/ with versioned gold_set/v1/.
| Decision | Choice |
|---|---|
| vLLM side-by-side | Path (a): CI OpenAI only; vLLM enforcement: manual in baseline + runbook |
| Live vs cassette | Hybrid: PR/smoke/fast = cassette ($0); full = optional live OpenAI |
| Gate | Absolute 3pp on CI-enforced metrics; exit code 1 (unit-tested) |
| Publish | web/public/benchmarks/extraction-quality-benchmark-data.json + bot suite |
See ADR-0066 and the gold-set README.
Success signals
- Gold set v1 ≥100 conversations, reviewed (Elshaday Mengesha, 2026-09-03)
-
run_eval.pyproduces per-category P/R + category-assignment + temporal metrics; CI on extraction/prompt paths - Baseline + 3pp absolute gate;
test_regression_gate.pyasserts exit 1 on >3pp drop - OpenAI CI numbers + vLLM manual block side-by-side in
baseline_results.json(enforcement labels)
Prerequisites
Edit on GitHub
Last updated on