Failure modes and recovery
IBEX separates authority failures from quality failures. Authority failures deny or return a typed dependency error. Quality failures may degrade, queue, or abstain, but cannot broaden visibility or invent a successful mutation.
Failure classes
| Dependency | Class | Required behavior |
|---|---|---|
| Auth, principal, policy | Authority | Deny or typed 503; no provider, context, or tool work |
| Budget/reservation | Authority/integrity | Deny or 503; never claim a hard budget that was not held |
| Redis on a protected path | Integrity/coordination | 503 under the protected profile; no permissive fallback |
| Context retrieval | Quality | Directive-only or verified hot-cache context; never wider scope |
| Embedder/reranker/model | Quality/async | Queue, fallback, quarantine, or abstain |
| Provider | External | Typed error, bounded retry, breaker, optional policy-compatible fallback |
| Evidence outbox | Integrity | Durable buffer or explicit uncertified outcome; never silent loss |
| ClickHouse/OTel | Projection | Buffer or degraded analytics; canonical evidence remains PostgreSQL |
| Object storage | Artifact | Block artifact-dependent action or use a verified retained reference |
| Queue/worker | Async | Retry, dead-letter, replay, and visible operation status |
Service-level behavior
Proxy
| Failure | Behavior |
|---|---|
ValidateToken timeout or outage | 503 SERVICE_DEGRADED; provider is not called |
ValidateAgent outage | 503 AUTH_UNAVAILABLE; provider is not called |
| Rate-limit Redis failure outside development | 503 SERVICE_DEGRADED before provider work |
| Org model-policy load failure | 503 SERVICE_DEGRADED; no ungoverned route |
| Context assembly timeout | Proceed with directive-only or history-only context and record fallback |
| ClickHouse unavailable | Drop or buffer traces; chat continues |
| Provider breaker open | 503 with Retry-After; optional eligible fallback |
The proxy must never call a provider after a security, policy, capability, reservation, or evidence-integrity denial.
Auth
/ready checks PostgreSQL and the gRPC listener. A missing policy-signing key makes IssuePolicySnapshot fail closed outside development. ValidateToken has no caller bearer and must remain on a trusted private network. If Redis is unavailable, revocation propagation and the auth cache are not silently treated as healthy.
Memory and context
Memory requires PostgreSQL/pgvector for durable reads and keyed writes; the embedder is required for vector writes. Redis is optional for hot-cache behavior. Context is a quality dependency: its failure must degrade to directive-only, not block inference.
Workers and evidence
Extraction, embeddings, deletion, billing reconciliation, and audit tasks are asynchronous. Queue delivery is not exactly-once. Every mutation needs an operation ID and idempotent effect. The relay is at-least-once by design; a projection cannot be treated as the authority for deletion or acknowledgement.
Readiness versus liveness
| Endpoint | Meaning | Must not mean |
|---|---|---|
/health | Process is alive | Dependencies are usable |
/ready | Selected profile's required dependencies are available | Every optional quality service is healthy |
/metrics | Telemetry is exposed | A measured SLO exists |
A process may be live and not ready. Readiness must not report healthy when security or evidence integrity is unavailable. Optional quality dependencies may be degraded and visibly reported.
Recovery classes
Recovery is planned per data class, not as one database restore:
| Class | Examples | Recovery rule |
|---|---|---|
| Canonical control | Organizations, users, agents, tokens, policies, reservations, operations, lifecycle | Restore first; authority must be consistent before serving |
| Evidence/outbox | Runs, events, spans, outbox rows | Restore with canonical state; replay must not resurrect tombstoned resources |
| Object artifacts | Redacted payloads, archives, model artifacts | Restore with metadata and retention/legal-hold state |
| Redis projections | Cache, hot memories, rate counters, leases | Rebuild; never treat as durable truth |
| Vector/search projections | Embeddings, HNSW, full-text | Rebuild from canonical rows and pinned embedding profile |
| Queues | Extraction and audit work | Replay only with idempotent effects; otherwise reconcile operation state |
A cache or index may be discarded and rebuilt. A deletion tombstone may not be discarded to “repair” a projection.
Migrations
The target sequence is:
expand → dual-read/write → backfill → verify → contract
Every migration needs lock and statement timeouts, throttling/backpressure, restart/resume behavior, abort criteria, and a roll-forward or rollback decision. Mixed-version compatibility must be tested. Embedding or tokenizer profile changes are versioned migrations, never silent replacement of an index.
The repository currently has PostgreSQL migrations through 000047_relay_fence_checker. Migration presence is not restore or HA evidence.
Shutdown and drain
On shutdown, services should:
- Stop accepting new admissions.
- Drain in-flight streams within the configured deadline.
- Cancel bounded work that cannot finish.
- Persist worker state and release or expire leases.
- Close dependencies after in-flight work completes.
Provider SSE and operator-event SSE have separate deadlines. Do not replace the ASGI server's SIGTERM handling with an incompatible custom handler.
What is not proven
- No tested RPO/RTO per data class.
- No hosted backup/restore drill is recorded.
- No HA failover, multi-zone, or capacity evidence is accepted.
- Helm is render/lint scaffolding only; it is not a deployable production topology.
- Object-store purge and full deletion certification remain open.
- Evidence replay has local PostgreSQL tests, not production ClickHouse or hosted recovery evidence.
Operator checklist
- Confirm
/readyreflects the dependencies required by the selected profile. - Confirm the failure is authority or quality before choosing deny versus degrade.
- Check whether a provider call happened before a deny.
- Preserve the request ID, trace ID, and operation ID; do not log tokens or content.
- For deletion or replay incidents, check tombstones before rebuilding projections.
- Record the profile, artifact digests, and limitations with any recovery result.
Related
- Health checks — per-service probes
- Incident response — severity and containment
- Evidence and relay — outbox and fencing
- Billing and budgets — reservation failures
- Kubernetes — why the chart is not a deployment guarantee
- Status and evidence — gate vocabulary