IBEX Harness
Phase 3 core memory substrate

find_similar read path using VectorStore.search() with HNSW index and GIN full-text fallback when vector results are sparse. EXPLAIN ANALYZE verification of index usage required.

Milestone 3.D.1 — Semantic search read path

Status: Complete
Goal: Track D — Read/Ranking Pipeline
Phase: 3 — Core Memory Substrate
Estimated effort: 2 days

Step 0 — Scoring reconciliation (handoff to 3.D.2)

Before implementing the read path, reconcile composite scoring state:

ItemFinding
Scoring moduleservices/memory/app/scoring/ (composite.py, half_life.py)
Category half-lifeAlready category-conditional in half_life.py (factual=180d … episodic=14d); multi-label uses shortest half-life
3.2.1 vs 3.D.2 page3.2.1 checklist is correct; the 3.D.2 milestone page described a bug that was fixed in Track B. Regression: test_old_factual_outranks_fresh_episodic_at_equal_relevance in test_composite.py
3.D.1 scopeMust not call composite_score() — raw similarity / ts_rank_cd only
3.D.2 remainingWire composite into search ranking, optional config-driven half-life table, doc updates

Ranking state for 3.D.2 (handoff)

TopicCurrent state
Ranking todayVector hits first by raw cosine similarity; FTS append by ts_rank_cd; no composite_score()
Score fieldMemorySearchResult.similarity holds raw similarity or FTS rank depending on source
Source values"vector" | "full_text"
Wire pointAfter merge/dedupe in repository.py find_similar() — apply composite_score() before final ordering
Half-lifeAlready category-conditional in app/scoring/half_life.py
Regression testtest_old_factual_outranks_fresh_episodic_at_equal_relevance in test_composite.py
Fallback interactionComposite scoring applies to merged vector+FTS candidates; preserve dedupe by memory_id

Delivered

  • MemoryReadRepository.find_similar() — vector primary, GIN supplemental when sparse
  • POST /v1/memories/search — memory:read, minimal request (agent_id, query, limit, min_similarity, min_confidence)
  • Fallback trigger: len(hydrated_vector_results) < limit — caller-relative, not a fixed constant (ADR-0058)
  • Merge: vector hits first (desc similarity), FTS append (desc ts_rank_cd), dedupe by id, cap at limit
  • Metric: ibex_memory_search_fallback_total{triggered} (increments only when FTS returns new candidates)
  • Toggle: IBEX_MEMORY_SEARCH_FALLBACK_ENABLED (default true)

Fallback trigger semantics

"Sparse" means the caller's page is unfilled after vector search + hydrate — whether because corpus < limit, min_similarity filtered ANN hits, or min_confidence dropped rows. There is no fixed VECTOR_FALLBACK_THRESHOLD; Phase 3 goals "vector results < K" means caller limit (K), not a magic number.

ScenariolimitConditionFTS fires?
Typical agent10 (default)15 memories, all pass filtersNo
Small agent108 memoriesYes
Large limit2015 memoriesYes
Full page3≥3 vector hits after hydrateNo

Latency (integration smoke, 500-row corpus, CI hardware)

  • Vector-only find_similar p95 gate: < 100ms (test_find_similar_latency_smoke; aligns with DEPLOYMENT alert threshold)
  • Phase 3 HNSW component budget: p95 < 30ms @ 1M vectors (latency table)
  • FTS supplement adds a small measured delta on the same smoke test; hard gate is vector-only p95

Schema (no new migration)

Confirmed in 000017_memory_schema_v2_expand.up.sql:

  • search_vector (generated tsvector) + idx_memories_search_vector (GIN)
  • idx_memories_embedding_hnsw (m=16, ef_construction=64)

Success signals

  • Semantic search uses the HNSW path (pg_stat idx_scan runtime gate in test_hnsw_index_scan_at_runtime)
  • GIN full-text fallback uses idx_memories_search_vector at runtime (test_gin_index_used_at_runtime: production full_text_search() + test-only explain_gin_probe_plan; HNSW uses pg_stat idx_scan in test_hnsw_index_scan_at_runtime)
  • Cross-org result leakage covered by integration isolation test
  • Full-text fallback triggers when vector results are sparse
  • Search calls are always org/agent scoped — no unscoped queries
  • Unit coverage ≥95% on app/read/ (100% at merge)

Prerequisites

  • Milestone 3.1.1 merged (HNSW index and search_vector GIN index in schema)
  • Milestone 3.2.1 merged (VectorStore ABC and PgVectorStore implementation available)
  • Track C milestones (write path in place for test fixtures)
Back to roadmap overview
Edit on GitHub

Last updated on

Previous3.C.4 Multi-label memory classificationNext3.D.2 Composite scoring in search ranking

On this page

0%