Intelligence

Extraction

Session-to-memory extraction, provider adapters, gold-set evaluation, and live vLLM limitations.

Extraction turns a completed session into candidate memories. It is a batch job, not an inline proxy step. The extraction path is implemented and locally evaluated; the live vLLM path is manual because CI has no GPU runner.

Provisional Ledger class: provisional
Verified 2026-10-21Cassette-backed evaluation; live vLLM is manual

Input contract

The extraction task receives:

  • org_id
  • agent_id
  • session_id
  • a list of turns with indices and content

The task returns status=ok or skipped for an intentional session gate. A skipped session is not an error, but it is also not an extraction result.

Provider modes

ModeProviderEvidence
openaiOpenAI-compatible chat completionCassette and unit tests in CI
vllmSelf-hosted OpenAI-compatible endpointUnit tests with httpx.MockTransport; real GPU run is manual

The model must return JSON matching BatchExtractionResult:

{"turns":[{"turn_index":0,"memories":[]}]}

No markdown fences. No prose wrapper. A schema violation must fail the task rather than persist malformed memory.

Memory writes

For each extracted memory, the worker posts to the memory service with:

  • the session ID
  • labels and optional temporal fields
  • a bearer token carrying memory:write

The worker must not log extracted content. Use memory IDs, turn indices, and task IDs for diagnosis.

Quality evaluation

The gold-set harness lives under services/worker/eval/:

cd services/worker
.venv/bin/pytest -q eval/
.venv/bin/python eval/run_eval.py --mode cassette
.venv/bin/python eval/regression_gate.py

The cassette path is the CI signal. The checked-in OpenAI baseline is oracle-aligned, so a perfect score proves scoring and gate wiring rather than live model quality. run_eval.py --mode vllm is a manual comparison with no published metrics, not a merge gate. Contract hashes for the prompt and schema are part of the evaluation contract. See Extraction evaluation.

Live vLLM checklist

  1. Start an OpenAI-compatible vLLM server with a model name that matches configuration.
  2. Set IBEX_WORKER_EXTRACTION_PROVIDER=vllm and IBEX_WORKER_EXTRACTION_BASE_URL.
  3. Set IBEX_WORKER_MEMORY_BASE_URL and a memory:write token.
  4. Enqueue a completed session.
  5. Confirm the task returns ok, the JSON matches the schema, memory receives one POST per memory, and trace rows contain counts but no prompt or completion text.

Limits

  • Mocked unit tests are not live integration evidence.
  • The evaluation gate is not a hosted quality guarantee, and the oracle-aligned 1.0 baseline is not live quality evidence.
  • Prompt and schema hashes do not prove extraction correctness.
  • Queue durability and exactly-once projection remain open.