Phase 4 operator platform

The operator runtime produces machine-verifiable contract, browser, security, resilience, performance, and rollback evidence.

Milestone 4.P.6 — Contract & Assurance Harness

Status: planned/specification — executable assurance gate, not a static smoke plan. Outcome: P6-bootstrap specifies and mounts the harness, manifest, parser, fixture, required-check, and rollback interfaces; P6-slice-certification records evidence per D slice. No global assurance completion is claimed by the harness specification alone. Track: Track P — Operator Readiness, Evidence & Control Plane Dependencies: P0 + P1 for P6-bootstrap. Each P6-slice-certification consumes the relevant D slice plus its own P contract; D1 consumes P6-bootstrap, while D2–D6 consume P6-slice-certification. Estimated effort: 3 weeks Gate: G6 assurance

P6-bootstrap

P6-bootstrap is the harness foundation: pinned-environment conventions, schema/parser interfaces, fixture versioning, artifact-manifest shape, required-check inventory, and rollback-record shape. It is a specification/mounting gate and does not assert that runtime, browser, resilience, or security checks have passed.

P6-slice-certification

For each D slice, P6-slice-certification runs only after that slice's named P contract is available. It attaches the slice-specific runtime, browser, tenant-negative, hostile-content, performance, resilience, and rollback outputs that actually ran. Missing or unverified outputs remain planned/specification or residual-risk; they are never inferred from the harness existing.

Executable assurance suite

CI must perform the following against a pinned environment and fixture version:

  • Capture and diff an OpenAPI snapshot, generate the TypeScript client, build it, and run old/new compatibility fixtures.
  • Parse every API response at runtime through schemas tied to the generated contract before browser assertions or DTO use; reject unknown/malformed security-sensitive shapes and record parser failures.
  • Seed four roles (viewer, operator, admin, owner) across two tenants and run authenticated Playwright journeys, including org isolation, revocation, step-up, lifecycle, evidence, query, cost, and operator SSE behavior.
  • Run axe and keyboard checks, deterministic visual baselines (fixed data/time/locale, reduced motion), and hostile-content/redaction rendering checks for traces, memories, directives, incidents, exports, and errors.
  • Prove cache isolation for authenticated API/RSC/export responses across users and tenants; test Cache-Control/key scope and no cross-tenant replay.
  • Exercise SSE reconnect, Last-Event-ID, duplicate suppression, slow consumers, write deadlines, bounded buffers, provider-stream separation, and graceful drain.
  • Run performance/load budgets with p50/p95/p99 latency, error rate, throughput, query cost, SSE lag, memory, queue lag, and explicit stop conditions.
  • Upload failure traces, screenshots, videos, JUnit, axe, visual diff, performance, security, and tenant-negative reports.

Artifact manifest and required checks

Each run writes an immutable manifest containing commit SHA, image/artifact digests, environment, fixture/schema versions, topology, test commands, timestamps, result, thresholds, and links to every artifact. The required-check inventory is generated from branch-protection rules and names the OpenAPI/client, runtime-parser, contract, Playwright, axe, visual, hostile-content, cache-isolation, SSE, performance, security, and Python/Semgrep checks. Missing or unregistered required checks fail closed.

Rollback assurance

The suite records the rollback command and prior digest/configuration, verifies schema compatibility, reruns runtime smoke and tenant-negative checks after rollback, and blocks promotion if rollback evidence is absent. An evidence bundle is not complete merely because a static shell loaded.

Findings addressed

F4-012, F4-028, F4-029, F4-030, F4-034; F4-035 remains a D0/E4 surface-separation and sign-off finding.

Exit checklist

  • OpenAPI snapshot, generated client, compatibility fixtures, and runtime parsing are required CI checks.
  • Four-role/two-tenant Playwright evidence runs against the operator topology.
  • axe, keyboard, visual, hostile-content, redaction, and cache-isolation artifacts are uploaded.
  • SSE resilience and performance thresholds are measured with stop conditions.
  • Artifact manifest is immutable and complete; required-check inventory matches protection rules.
  • Rollback transcript and post-rollback tenant/security smoke are attached.
Edit on GitHub

Last updated on