P6-backed load, degradation, SSE, recovery, and rollback evidence demonstrates declared reliability guarantees.
Milestone 4.E.2 — Reliability, Resilience & Recovery Gate
Status: planned/specification — consumes resilience evidence; no resilience result is claimed here. Outcome: Declared SLO, RPO, RTO, cost, resilience, and rollback guarantees are measured in staging. Track: Track E — Release Readiness & Phase Exit Dependencies: resilience evidence from 4.C.5 and P5, plus the relevant P6-slice-certification outputs. E2 does not depend on E1; E3 follows E1 and E2. Estimated effort: 3 weeks Gate: G2 resilience
Scope and experiments
Run sustained operator query/SSE load with p50/p95/p99, error rate, throughput, cost, memory, queue lag, and SSE lag thresholds. Exercise one-byte-per-second consumers, bounded buffers, write deadlines, shedding, reconnect/resume, duplicate suppression, and drain. Inject provider mid-stream failure, Redis/ClickHouse/Postgres degradation, queue saturation, slow consumers, pod kill, and network partition. Execute backup/restore and rollback with tenant isolation, evidence/outbox position, achieved RPO/RTO, and immutable digest/configuration records.
Each experiment records workload, threshold, observation window, error budget, tenant invariants, degraded mode, stop condition, owner, and P6-slice-certification artifact links. Missing data is labeled incomplete, never zero.
Verification and rollback
The gate requires applicable P6-slice-certification artifacts to be complete and parser-valid, no cache cross-talk, no SSE memory growth under the declared soak, post-restore tenant isolation, and rollback service within declared RTO. Failed experiments create owned findings and halt promotion.
Trigger: SLO breach, restore failure, or rollback failure. Reversal: halt promotion, disable the operator origin or route to the last known-good digest, and preserve the evidence bundle. Verify: rerun the affected P6 check and append the finding disposition.
Findings addressed
F4-007, F4-011, F4-020, F4-033.
Exit checklist
- Applicable P6-slice-certification performance and slow-client reports meet declared thresholds.
- SSE reconnect/resume, shedding, and drain evidence is attached.
- Provider/dependency degradation and queue saturation are recorded.
- Restore report includes achieved RPO/RTO and tenant-negative checks.
- Chaos/rollback transcript uses an immutable digest and links the P6-slice-certification manifest.
- P5 open platform evidence is implemented-with-evidence or accepted as residual-risk.
Last updated on