Milestone 3.5.B.3 — Incremental Turn Tracking and IdempotencyCopy Anchor Link

Status: Completed Goal: Track B — Extraction Pipeline
Phase: 3.5 — Extraction & Context Assembly
Estimated effort: 2 days
Track: Track B — Extraction Pipeline
Depends on: 3.5.B.2
ADR: ADR-0065

---

Why This Milestone ExistsCopy Anchor Link

3.5.B.2 writes extracted memories over HTTP to services/memory and updates last_extracted_turn in a separate worker Postgres UPDATE afterward. A worker-local Postgres transaction cannot roll back memory rows another service already committed — the original “single async with db.transaction()” sketch is architecturally impossible.

Pre-3.5.B.3, X-Idempotency-Key was derived from LLM output content. Retries with different wording produced different keys and different content_hash values, allowing duplicate memories and wasted extraction spend.

This milestone closes that gap with identity-based HTTP idempotency, mandatory SessionStore, and per-turn pointer advances. Content-hash exact dedup (m3.C.2) remains a complementary backstop, not a substitute.

---

Non-GoalsCopy Anchor Link

• Distributed two-phase commit / outbox between worker and memory service
• Batch write API on services/memory
• Changes to memory-service dedup algorithms
• Quality evaluation (3.5.B.4)
• Operator-dashboard-visible turn-level status tracking (Phase 4)
• Moving last_extracted_turn mutation behind a new memory-service HTTP endpoint
• Changing worker→memory PAT auth

---

Design (as implemented)Copy Anchor Link

1. Identity-based idempotency keys — X-Idempotency-Key = uuid5(ns, sha256(f"{org_id}:{session_id}:{turn_index}:{memory_ordinal}")). Does not include LLM output text or a batch content fingerprint.
2. Memory HTTP retry semantics — treat 409 IDEMPOTENCY_CONFLICT and 409 DUPLICATE_CONTENT as definitive success (first-commit wording wins); IDEMPOTENCY_IN_PROGRESS and 5xx remain retryable.
3. Mandatory SessionStore — extract_session_memories fails closed without database_url.
4. Per-turn pointer advance — after each turn's memories succeed, UPDATE last_extracted_turn = GREATEST(...) for that turn index. Mid-batch crashes keep completed turns skipped on retry.

Residual limitation (documented): if a retry returns a different count of memories for the same turn, ordinals can misalign. Exact/near-dedup is the backstop. Crash after writes but before that turn's pointer UPDATE may incur one additional provider.extract — bounded, not zero LLM cost.

See ADR-0065 for full rationale.

---

Success signalsCopy Anchor Link

• Same turn+ordinal, different wording → same X-Idempotency-Key (regression vs content-derived keys)
• Crash mid-batch → pointer only for fully completed turns; retry safe for committed slots
• extract_session_memories run twice in immediate succession → second run noop (pointer unchanged, no extra writes)
• 409 IDEMPOTENCY_CONFLICT and 409 DUPLICATE_CONTENT treated as non-fatal success
• Missing database_url → hard failure (SessionStore mandatory)

---

PrerequisitesCopy Anchor Link

• 3.5.B.2 Cost-tiered extraction execution