Phase 3 core memory substrate

Phase 3 — Goals

Core Memory Substrate: schema, vector store, write pipeline, read/ranking, and exit gate.

Phase 3 — Goals

Track A — Memory Schema & Data Foundation

Goal 3.A: Establish the memory schema with HNSW, temporal validity, and multi-label taxonomy

Milestones:

  • 3.1.1 — Postgres schema v2 (completed): expand foundation ibex_core.memories (from G5.M1 / ADR-0047) with HNSW vector index (m=16, ef_construction=64), quality columns, generated search_vector, bounded embedding_model/metadata CHECKs. Labels + primary sync already shipped in ADR-0048; expand ADR is ADR-0052 (not ADR-0040).

Acceptance criteria:

  • Migration applies cleanly from zero via make db-migrate
  • HNSW index type confirmed in CI assertion (not just present — type-checked)
  • RLS integration test proves cross-org denial on memories (labels covered in 2.5.G5.M2)
  • Primary-category sync verified via existing trigger (insert 2 labels, differing confidence, correct one wins)

Track B — Vector Store & Embedding Integration

Goal 3.B: Pluggable vector store with category-conditional composite scoring

Milestones:

  • 3.2.1 — VectorStore interface + pgvector-HNSW implementation: Shipped — VectorStore ABC, PgVectorStore, composite scoring v2, embedding client, HNSW benchmarks (ADR-0053).

Acceptance criteria:

  • VectorStore ABC with PgVectorStore implementation, ef_search set per-transaction
  • Composite scoring v2 unit-tested against documented formula
  • Benchmark run at 10K/100K rows committed (1M full-profile optional per ADR-0061)
  • embedding_client.py has zero direct ML library imports

Track C — Memory Write Pipeline

Goal 3.C: 9-step write pipeline with PII detection, dedup, and temporal conflict resolution

Milestones:

  • 3.C.1 — PII detection and redaction: Shipped — Presidio two-tier detection, redaction before embed (ADR-0054).
  • 3.C.2 — Content and near-duplicate detection: Shipped — exact + near-dup dedup (ADR-0055).
  • 3.C.3 — Temporal-interval-aware conflict detection: Shipped — interval overlap + sequential supersede (ADR-0056).
  • 3.C.5 — Write-path persistence & orchestration: Shipped — ADR-0057.
  • 3.C.4 — Multi-label memory classification: Shipped — ADR-0048 (extraction-worker prompt is Phase 3.5).

Acceptance criteria:

  • Sequential-fact fixture classified as supersedes with zero LLM calls
  • Genuinely overlapping-interval contradiction still escalates to LLM correctly
  • PII detection: >95% recall on structured PII labeled fixture
  • Redaction happens strictly before embedding (pipeline ordering test)

Track D — Read/Ranking Pipeline & Hot Cache

Goal 3.D: Semantic search, composite scoring, and hot cache

Milestones:

  • 3.D.1 — Semantic search read path: Shipped — VectorStore.search(), FTS fallback (ADR-0058).
  • 3.D.2 — Composite scoring in search ranking: Shipped — composite ordering in search (ADR-0053).
  • 3.D.3 — Hot cache (Redis sorted set): Shipped — MemoryHotCacheReader, atomic top-50 trim (ADR-0059).

Acceptance criteria:

  • find_similar uses HNSW index (confirmed via EXPLAIN ANALYZE)
  • Cross-org result leakage impossible (integration test)
  • Hot cache score formula reuses CATEGORY_HALF_LIFE_DAYS

Track E — Exit Gate

Goal 3.E: Prove the composed system enforces isolation and meets quality bars

Milestones:

  • 3.E.1 — Memory security integration suite (TestMemory_ISO_*): Shipped — 12 ISO cases, memory-security-integration CI, Tier-1 read-path PII guard.
  • 3.E.2 — End-to-end memory lifecycle test: Shipped — make e2e-smoke-p3-memory, e2e-smoke-p3-memory CI job.
  • 3.E.3 — Retrieval-quality benchmark sign-off: Shipped — HNSW + ranking + write CI regression gates (ADR-0061).
  • 3.E.4 — ADR consolidation: Shipped — ADR-0060, ADR-0061; phase register/index reconciled (decisions).
  • 3.E.5 — Phase 3 exit audit: Shipped — gap register, summary; zero open P0 gaps.

Acceptance criteria:

  • All ISO-* cases pass with real containers
  • memory-security-integration CI job required on main
  • Benchmark baseline committed; CI fails on regression
  • Zero open P0 gaps in exit audit
Edit on GitHub

Last updated on