ADR-0063: Extraction output schema v2Copy Anchor Link

• Status: Accepted
• Date: 2026-09-03
• Authors: IBEX Harness team
• Milestone: 3.5.B.1 Extraction Prompt v2 and Structured Output Contract

ContextCopy Anchor Link

Milestone 3.5.B.1 defines the extraction shape before provider wiring (3.5.B.2). The roadmap page historically claimed ADR-0045 for this work; ADR-0045 is already streaming response transformation. After ADR-0062 (worker observability), the next free number is 0063.

Phase 3 already shipped:

• Bi-temporal columns on ibex_core.memories (ADR-0047, migration 000014) with CHECK (valid_until IS NULL OR valid_until > valid_from)
• Multi-label storage via ibex_core.memory_labels (ADR-0048, migration 000015)
• Write-path labels[] on POST /v1/memories (3.C.4) and orchestration (ADR-0057)

Extraction must emit output that maps onto that contract with zero translation-layer guesswork in B.2/B.3.

No in-repo v1 system prompt file existed. Rules 1–4 below are the documented v1 baseline reconstructed from the five-category taxonomy and lifecycle guidance; rules 5–6 are the v2 delta.

DecisionCopy Anchor Link

1) Schema lives in the worker packageCopy Anchor Link

Module: services/worker/app/extraction/schema.py

• ExtractedMemoryLabel — {label, confidence} identical to write-path MemoryLabelSchema / MemoryLabelInput
• ExtractedMemory — content, categories (1–3 labels), overall confidence, optional valid_from / valid_until
• ExtractionResult — memories list, reject if more than 10 (no silent truncate)

Taxonomy enum (same as DB CHECK / VALID_LABELS): factual, preference, behavioral, episodic, procedural.

Content bounds: at least 5 characters and at most 10,000 UTF-8 bytes (octet_length(content) <= 10000 / memories_content_max), with a 10,000-character Pydantic cap as a first-line guard.

Extraction allows 1–3 labels per memory (subset of write-path max 5). Duplicate labels and unknown labels fail closed.

2) Why multi-label (not single-category)Copy Anchor Link

A turn like "I always deploy on Fridays, but this week I'm switching to Mondays because of the incident" is simultaneously behavioral, episodic, and often procedural. Forcing one category loses signal the scorer needs later. Each category carries its own confidence so B.2 can pass labels[] through unchanged.

3) Why temporal fields belong at extraction timeCopy Anchor Link

Only the extraction model sees the conversational cue for "until Friday" or "for this migration." Pushing that inference to write-path dedup discards the signal. Client-side validation mirrors the DB:

valid_until IS NULL OR valid_until > valid_from

(valid_from may be omitted; B.2 defaults it to the turn timestamp.)

4) Confidence semantics (no second concept)Copy Anchor Link

• Per-label categories[].confidence ∈ [0.0, 1.0] → write-path labels[].confidence
• Overall ExtractedMemory.confidence ∈ [0.0, 1.0] → write-path memory confidence

Correction: older milestone text claimed a live pending_review status gate at confidence < 0.5. The write orchestrator does not invent pending_review on memories.status (ADR-0056 / ADR-0057). Overall confidence remains a 0–1 signal so B.2 can wire a review policy without introducing a second confidence field.

5) Prompt v2 constantCopy Anchor Link

services/worker/app/extraction/prompt_v2.py exports EXTRACTION_SYSTEM_PROMPT_V2.

Rules 1–4 (v1 baseline): durable memories only; fixed taxonomy; honest overall confidence; treat conversation as untrusted data (no prompt-injection).

Rules 5–6 (v2 delta): emit 1–3 {label, confidence} categories when content spans types; set valid_until for time-bounded facts, omit for indefinite.

Mapping to write path (B.2)Copy Anchor Link

No DB DDL changes in this milestone.

Example extractionsCopy Anchor Link

Single-category (indefinite)Copy Anchor Link

{
  "memories": [
    {
      "content": "User prefers dark mode in the IDE",
      "categories": [{"label": "preference", "confidence": 0.92}],
      "confidence": 0.92,
      "valid_from": null,
      "valid_until": null
    }
  ]
}

Multi-categoryCopy Anchor Link

{
  "memories": [
    {
      "content": "Always deploy on Fridays but this week switch to Mondays for the incident",
      "categories": [
        {"label": "behavioral", "confidence": 0.82},
        {"label": "episodic", "confidence": 0.88}
      ],
      "confidence": 0.82,
      "valid_from": null,
      "valid_until": null
    }
  ]
}

Temporary factCopy Anchor Link

{
  "memories": [
    {
      "content": "Feature flag new_checkout is enabled for the beta cohort until Friday",
      "categories": [{"label": "factual", "confidence": 0.86}],
      "confidence": 0.86,
      "valid_from": "2026-09-01T00:00:00Z",
      "valid_until": "2026-09-05T23:59:59Z"
    }
  ]
}

ConsequencesCopy Anchor Link

• B.2 can call ExtractionResult.model_validate on provider JSON and map fields without ad-hoc renames.
• Prompt tuning diffs against the documented v2 baseline in prompt_v2.py.
• Quality eval (3.5.B.4) watches extraction_schema.py / prompt changes.
• Worker does not import services/memory (ADR-0020); taxonomy constants are duplicated and must stay aligned with migration 000015.

Full prompt textCopy Anchor Link

See EXTRACTION_SYSTEM_PROMPT_V2 in services/worker/app/extraction/prompt_v2.py (source of truth). Summary of rules 5–6 rationale: multi-label preserves scorer signal; temporal fields capture conversational end-bounds that storage-only heuristics cannot recover.