ADR-0058: Semantic search FTS fallback triggerCopy Anchor Link

• Status: Accepted
• Date: 2026-08-29
• Authors: IBEX Harness team
• Milestone: 3.D.1 Semantic search read path

ContextCopy Anchor Link

Milestone 3.D.1 ships MemoryReadRepository.find_similar() with HNSW vector search as the primary path and GIN ts_rank_cd full-text search as a supplement when vector results do not fill the caller's requested page.

Phase 3 goals describe fallback when "vector results < K". During implementation review, a fixed sparse threshold (decoupled from caller limit) was considered to avoid running FTS when an agent's corpus is smaller than the requested page size.

Options ConsideredCopy Anchor Link

1) Fixed sparse threshold (e.g. supplement only when len(vector_results) < 3)Copy Anchor Link

Pros: Avoids FTS on agents whose corpus is smaller than limit but still semantically well-covered by vector hits.

Cons: Misses min_confidence-driven sparsity (vector ANN returns candidates that hydrate filters out). Fires FTS when vector already returned ≥ limit hits but a fixed threshold is low. Adds a magic constant unrelated to caller intent.

2) Caller-relative trigger: len(hydrated_vector_results) < limit (chosen)Copy Anchor Link

Pros: "Sparse" means the caller's page is unfilled — regardless of whether the shortfall is due to corpus size, min_similarity filtering, or min_confidence dropping rows during hydrate. No magic constant. Matches API contract: return up to limit best results.

Cons: When limit exceeds corpus size (e.g. limit=20, 15 memories), FTS runs on every search for that agent. Measured FTS overhead at integration scale is small (see Consequences).

DecisionCopy Anchor Link

1. Trigger: settings.search_fallback_enabled AND is_searchable_query(query_text) AND len(hydrated_vector_results) < limit.
2. No fixed VECTOR_FALLBACK_THRESHOLD constant — the milestone sketch constant was illustrative, not a design requirement.
3. Merge: Dedupe by memory_id (vector wins). Hydrate scoring fields, apply composite_score(), sort descending, cap at limit. HTTP similarity remains the retrieval metric (cosine or raw ts_rank_cd); ordering reflects composite. FTS supplemental hits use relevance sentinel 0.5 for composite scoring only (3.D.2).
4. Metric: ibex_memory_search_fallback_total{triggered="true"} increments only when FTS returns at least one new candidate (not on every trigger evaluation).
5. Toggle: IBEX_MEMORY_SEARCH_FALLBACK_ENABLED (default true).

Index verification (Semgrep relocation)Copy Anchor Link

• HNSW runtime gate: PgVectorStore.search + pg_stat idx_scan on idx_memories_embedding_hnsw (test_hnsw_index_scan_at_runtime). EXPLAIN helpers remain in benchmarks/memory/plan_explain.py for hnsw_bench only.
• GIN verification: test-only probe transaction that drops competing partial btree indexes (restored on rollback), disables RLS, and applies planner hints (explain_gin_probe_plan + assert_gin_index_used) plus production full_text_search() hit count in test_gin_index_used_at_runtime. Partial btree indexes sharing the probe predicate otherwise route @@ through heap filters without exercising the GIN index.

ConsequencesCopy Anchor Link

Frequency (typical agents)Copy Anchor Link

With default limit=10, agents holding 10–30 memories do not run FTS on every search unless filters shrink results below limit.

LatencyCopy Anchor Link

Phase 3 budgets (latency table):

Operational alert (DEPLOYMENT.md): memory search p95 > 100ms for 10 minutes.

Integration smoke test (test_find_similar_latency_smoke) seeds 500 rows and asserts vector-only find_similar p95 < 100ms on CI hardware (not 1M scale). FTS supplement adds a small delta measured in the same test; the hard gate is vector-only p95.

3.D.2 (shipped)Copy Anchor Link

Search ranking uses composite_score() after merge/dedupe and hydrate in find_similar() (app/read/ranking.py). FTS supplemental rows use fixed relevance 0.5 for the composite relevance component only; response similarity is unchanged.

RelatedCopy Anchor Link

• ADR-0053 — VectorStore ABC and HNSW SQL
• ADR-0057 — write path (separate broad-exception cleanup tracked in GitHub #634)