Phase 3 core memory substrate
Phase 3 — Goals
Core Memory Substrate: schema, vector store, write pipeline, read/ranking, and exit gate.
Phase 3 — Goals
Track A — Memory Schema & Data Foundation
Goal 3.A: Establish the memory schema with HNSW, temporal validity, and multi-label taxonomy
Milestones:
- 3.1.1 — Postgres schema v2 (completed): expand foundation
ibex_core.memories(from G5.M1 / ADR-0047) with HNSW vector index (m=16, ef_construction=64), quality columns, generatedsearch_vector, boundedembedding_model/metadataCHECKs. Labels + primary sync already shipped in ADR-0048; expand ADR is ADR-0052 (not ADR-0040).
Acceptance criteria:
- Migration applies cleanly from zero via
make db-migrate - HNSW index type confirmed in CI assertion (not just present — type-checked)
- RLS integration test proves cross-org denial on
memories(labels covered in 2.5.G5.M2) - Primary-category sync verified via existing trigger (insert 2 labels, differing confidence, correct one wins)
Track B — Vector Store & Embedding Integration
Goal 3.B: Pluggable vector store with category-conditional composite scoring
Milestones:
- 3.2.1 — VectorStore interface + pgvector-HNSW implementation: Shipped —
VectorStoreABC,PgVectorStore, composite scoring v2, embedding client, HNSW benchmarks (ADR-0053).
Acceptance criteria:
-
VectorStoreABC withPgVectorStoreimplementation,ef_searchset per-transaction - Composite scoring v2 unit-tested against documented formula
- Benchmark run at 10K/100K rows committed (1M full-profile optional per ADR-0061)
-
embedding_client.pyhas zero direct ML library imports
Track C — Memory Write Pipeline
Goal 3.C: 9-step write pipeline with PII detection, dedup, and temporal conflict resolution
Milestones:
- 3.C.1 — PII detection and redaction: Shipped — Presidio two-tier detection, redaction before embed (ADR-0054).
- 3.C.2 — Content and near-duplicate detection: Shipped — exact + near-dup dedup (ADR-0055).
- 3.C.3 — Temporal-interval-aware conflict detection: Shipped — interval overlap + sequential supersede (ADR-0056).
- 3.C.5 — Write-path persistence & orchestration: Shipped — ADR-0057.
- 3.C.4 — Multi-label memory classification: Shipped — ADR-0048 (extraction-worker prompt is Phase 3.5).
Acceptance criteria:
- Sequential-fact fixture classified as
supersedeswith zero LLM calls - Genuinely overlapping-interval contradiction still escalates to LLM correctly
- PII detection: >95% recall on structured PII labeled fixture
- Redaction happens strictly before embedding (pipeline ordering test)
Track D — Read/Ranking Pipeline & Hot Cache
Goal 3.D: Semantic search, composite scoring, and hot cache
Milestones:
- 3.D.1 — Semantic search read path: Shipped —
VectorStore.search(), FTS fallback (ADR-0058). - 3.D.2 — Composite scoring in search ranking: Shipped — composite ordering in search (ADR-0053).
- 3.D.3 — Hot cache (Redis sorted set): Shipped —
MemoryHotCacheReader, atomic top-50 trim (ADR-0059).
Acceptance criteria:
-
find_similaruses HNSW index (confirmed viaEXPLAIN ANALYZE) - Cross-org result leakage impossible (integration test)
- Hot cache score formula reuses
CATEGORY_HALF_LIFE_DAYS
Track E — Exit Gate
Goal 3.E: Prove the composed system enforces isolation and meets quality bars
Milestones:
- 3.E.1 — Memory security integration suite (
TestMemory_ISO_*): Shipped — 12 ISO cases,memory-security-integrationCI, Tier-1 read-path PII guard. - 3.E.2 — End-to-end memory lifecycle test: Shipped —
make e2e-smoke-p3-memory,e2e-smoke-p3-memoryCI job. - 3.E.3 — Retrieval-quality benchmark sign-off: Shipped — HNSW + ranking + write CI regression gates (ADR-0061).
- 3.E.4 — ADR consolidation: Shipped — ADR-0060, ADR-0061; phase register/index reconciled (decisions).
- 3.E.5 — Phase 3 exit audit: Shipped — gap register, summary; zero open P0 gaps.
Acceptance criteria:
- All ISO-* cases pass with real containers
-
memory-security-integrationCI job required onmain - Benchmark baseline committed; CI fails on regression
- Zero open P0 gaps in exit audit
Edit on GitHub
Last updated on