Intelligence

Extraction evaluation

The versioned gold set, cassette replay, regression gate, and why a perfect score is not a quality claim.

Provisional Ledger class: provisional

Verified 2026-10-21The harness and CI gate are implemented; live model quality is not established by the oracle-aligned baseline

The worker has a versioned extraction-quality harness. It is valuable because it makes prompt, schema, and scoring regressions visible. It is dangerous to read as a model benchmark: the CI baseline currently replays oracle-aligned cassettes, so a perfect score proves the scoring and gate wiring, not that a live model extracts perfectly.

Gold set v1

Property	Value
Version	`v1`
Conversations	125
Expected memory rows	130
Categories	Factual, procedural, preference, behavioral, episodic
Multi-label memories	10
Supersession cases	10
Label review	Hand-labeled synthetic conversations, reviewed 2026-09-03

Files separate inputs, expectations, provider cassettes, and baselines:

File	Role
`conversations.jsonl`	Conversation turns
`expected_memories.jsonl`	Expected memories plus temporal kind
`openai_cassettes.jsonl`	Deterministic provider output per conversation
`cassette_manifest.json`	Prompt/schema hashes, cassette kind, file hashes, counts
`baseline_results.json`	Per-provider metrics and enforcement mode

A future v2 can add cases without silently breaking comparability; comparisons must name the gold-set version.

Scoring

The harness computes, per category:

Precision and recall against expected memories
Category-assignment accuracy
Temporal-field accuracy
Macro precision and recall

It also tracks multi-label and supersession behavior. A memory that matches content but has the wrong category or temporal kind is not a full success.

Modes and enforcement

Provider	Enforcement	How numbers are produced
OpenAI `gpt-4o-mini`	`ci`	Cassette replay on PR/smoke/fast; optional live weekly run with a key
vLLM `Qwen2.5-14B-Instruct`	`manual`	No GPU CI runner; null metrics until a manual runbook run is published

Cassette replay is free and deterministic. The cassette_kind is provenance, not cryptographic proof of model output:

oracle_aligned_expected_json — cassettes embed the expected JSON, producing a perfect baseline while testing scoring and gate wiring.
live_openai_recorded — cassettes were recorded from a live OpenAI call.

The checked-in OpenAI baseline is oracle-aligned and reports 1.0 across gated metrics. The vLLM baseline has null metrics and enforcement: manual. Treat both honestly: one tests plumbing, the other is not yet measured.

Contract and oracle gates

Prompt and schema changes are contract changes. The harness hashes:

sha256 of the batch extraction system prompt
sha256 of the canonical BatchExtractionResult JSON schema

Cassette, smoke, and fast runs fail on drift unless a local override is set; CI must never set that override. If cassettes are still oracle-aligned and a pull request touches the prompt or schema, CI fails unless the PR body carries the reviewer-visible EXTRACTION_EVAL_ORACLE_OK=1 exception or the cassettes were re-recorded live. Bumping a hash alone does not satisfy the gate.

Regression gate

The gate compares the latest run to the CI-enforced provider block and fails on an absolute drop greater than 3 percentage points in any gated metric. Only enforcement: ci providers can fail the build. An empty or unmatched provider set is a failure, not a pass.

cd services/worker
.venv/bin/pytest -q eval/
.venv/bin/python eval/run_eval.py --mode cassette
.venv/bin/python eval/regression_gate.py

Outputs are written under eval/output/; the gate exits 1 on regression.

Recording live cassettes

export OPENAI_API_KEY=…       # budgeted; never commit
export EXTRACTION_EVAL_MODE=record
.venv/bin/python eval/run_eval.py --mode record

This refreshes the cassette file and manifest, sets cassette_kind=live_openai_recorded, and updates prompt/schema hashes. Re-run cassette evaluation and update the baseline metrics and refresh date. A live recording is a new evidence event; it does not retroactively make old numbers live.

Manual vLLM comparison

After the manual vLLM server is running:

export IBEX_WORKER_EXTRACTION_PROVIDER=vllm
export IBEX_WORKER_EXTRACTION_BASE_URL=http://127.0.0.1:8000/v1
cd services/worker
.venv/bin/python eval/run_eval.py --mode vllm

Copy metrics into providers.vllm.metrics, set last_refreshed, and keep enforcement: manual. Do not flip the mode to ci without a GPU-backed CI runner and accepted cost/availability evidence.

What the numbers do not prove

That live model precision or recall is 1.0.
That synthetic conversations represent customer traffic.
That extraction is safe against prompt injection or poisoned memory content.
That vLLM is benchmarked, GPU capacity exists, or latency fits the queue budget.
That a passing gate covers embedding quality, retrieval quality, or end-to-end answer quality.

Evaluation is one gate in the intelligence plane. Retrieval ranking, conflict handling, and packer quality need their own evidence.

Extraction — provider adapters and queue behavior
Embedder — embedding profiles and readiness
Workers — task families and retries
Ranking and packing — context quality boundaries
Roadmap and gates — where evaluation evidence fits

Edit this pageorReport an issue

Embedder

Embedding profiles, backend geometry checks, tenant-scoped caching, and readiness boundaries.

API reference

HTTP and gRPC contracts for the proxy, management plane, auth service, and health surfaces.