Milestone 4.D.4 — Failures, Incidents & Evidence BundlesCopy Anchor Link

Status: Planned and blocked. Until the hard prerequisites and acceptance evidence below exist, incident views are read-only and all incident state, assignment, comment, and bundle writes are disabled. The repository baseline is 8f8e130 plus pre-existing review documents; no implementation is claimed.

Outcome: Operators can triage tenant-scoped failures, and—only after the gate—manage a real incident state machine and produce signed, redacted, recoverable evidence bundles.

Track: Track D — Operator Capability Slices

Dependencies: 4.D.2; 4.P.1 identity/permissions/step-up; 4.P.2 event and evidence contracts; 4.P.3 audit, redaction, retention, and bundle contract; 4.P.4 query; signing/key-rotation and recovery ownership decisions.

Estimated effort: 3 weeks after prerequisites are accepted.

PurposeCopy Anchor Link

Preserve enough trustworthy, redacted evidence to reconstruct an incident after ordinary telemetry expires, without treating raw traceback or client-rendered payloads as a safe workflow.

ScopeCopy Anchor Link

• Read-only failure projection with filters by status, severity, owner, and time, plus safe hostile-payload rendering.
• Explicit incident state machine, severity, dedupe key, owner/assignment, comments/activity, linked traces/sessions/requests, and immutable transition history.
• Stable evidence selection, redaction state, digest, signature, retention/legal-hold metadata, actor, request ID, and bundle manifest.
• Recovery and failure paths for bundle creation, signing, storage, download, verification, expiry, and key rotation.

Non-goalsCopy Anchor Link

• No incident writes, assignment, comments, state transitions, or bundle downloads before the write gate is approved.
• No claim that failed_tasks is an incident state machine, that a UI status is durable, or that an unsigned export is postmortem evidence.
• No raw traceback, secret, provider credential, prompt, memory, or cross-tenant payload exposure by default.
• Notifications, paging integrations, automated remediation, and a full postmortem editor are later integrations.

Hard prerequisitesCopy Anchor Link

1. Incident states, legal transitions, terminal/reopen semantics, dedupe identity, concurrency behavior, and idempotency are specified.
2. The centralized backend command pipeline is available for authorization, tenant resolution, kill switch, fresh step-up, idempotency, validation, mutation/outbox, ledger, and audit.
3. Bundle contract specifies stable evidence selection, redaction classifications, digest/signature algorithm and key ownership, retention/legal hold, download authorization, and verification behavior.
4. Recovery owner and tested procedures exist for partial bundle creation, outbox lag, object-store failure, signing failure, key rotation, corrupt bundle, and failed restore.
5. Cross-tenant hostile-content tests and durable audit/action-ledger storage are available.

DeliverablesCopy Anchor Link

1. Read-only failure projection and incident list/detail DTOs with completeness, freshness, retention, and source-watermark state.
2. A server-enforced incident state machine with idempotent, audited transitions and durable action history.
3. A redaction-first bundle builder with manifest, digest, signature, retention/legal-hold metadata, and authorized verification/download.
4. Recovery runbook and failure-injection suite covering incomplete, expired, corrupted, unsigned, and unverifiable bundles.
5. Feature flag/kill switch that keeps all incident writes and bundle downloads disabled until the completion rule is satisfied.

Acceptance evidence (machine-verifiable)Copy Anchor Link

• State-machine contract tests enumerate allowed/forbidden transitions, concurrent updates, retries, dedupe, terminal states, reopen policy, and immutable history.
• Org A/Org B tests prove failure, incident, comment, assignment, audit, and bundle identifiers cannot cross tenants; missing organization context fails closed.
• Command-pipeline tests prove every write checks permission, tenant ownership, kill switch, fresh step-up, idempotency, and approval policy before mutation/outbox and ledger append.
• A golden incident fixture produces a manifest whose selected evidence, redaction decisions, actor, request ID, retention, digest, and signature verify from a fresh process after normal telemetry is unavailable.
• Negative tests prove secrets, raw traceback, forbidden prompt/memory/provider fields, and cross-tenant records are absent; hostile payloads render inertly.
• Failure-injection tests show safe retry/parking, no false success, recovery/restore, key-rotation verification, and read-only rollback. CI fails if the incident write flag is enabled without these artifacts.

Completion ruleCopy Anchor Link

Complete only when the state machine, signed/redacted bundle contract, recovery procedures, and failure evidence are implemented and machine-verifiably tested. Until then, incident surfaces remain read-only/write-disabled. Completion also requires a demonstrated rollback to read-only that preserves existing bundle verification and immutable history.

PR sequenceCopy Anchor Link

1. Freeze read-only failure/incident DTOs, data classification, source-watermark semantics, and the explicit disabled-write UI state.
2. Specify and implement the state machine, durable transition history, command pipeline integration, and cross-tenant/idempotency tests.
3. Implement redaction-first bundle manifests, digest/signature verification, retention/legal-hold metadata, and authorized download.
4. Add object-store/signing failure injection, recovery/restore, key rotation, corruption, expiry, and safe parking evidence.
5. Run an independent security/recovery review; enable writes only behind the gate, otherwise keep the kill switch closed.

Open questionsCopy Anchor Link

• Which incident states and transitions are canonical, and who may reopen or close a terminal incident?
• Which signing key service, algorithm, rotation cadence, and verification lifetime are approved?
• What evidence fields are retained under legal hold, and how are deletion requests represented without breaking bundle verification?
• What is the recovery objective and owner for incomplete or unavailable bundles?
• Which notification integrations, if any, may consume incident transitions after this milestone?

Rollout and rollbackCopy Anchor Link

Trigger: invalid transition, cross-tenant result, redaction/signature failure, unrecoverable bundle, or ledger inconsistency. Reversal: close the incident-write and download flags; serve read-only triage and verify existing bundles offline. Verify: no new state mutations succeed, queued commands are parked, immutable history remains readable, and recovery evidence passes before re-enabling writes.

Findings addressedCopy Anchor Link

F4-008, F4-011.

Exit checklistCopy Anchor Link

• Read-only mode and disabled-write behavior are evidenced.
• State-machine, tenant, permission, idempotency, and ledger tests are attached.
• Signed/redacted bundle verification and recovery/failure evidence are attached.
• Rollback to read-only is demonstrated; no write enablement is claimed prematurely.