Phase 4 operator platform
Phase 4 — Findings Register
Evidence-backed findings that shaped the Track P prerequisite and redesigned Track D and Track E roadmap.
Phase 4 — Findings Register
Executive finding
The original Phase 4 plan was a minimal feature list. It described screens and endpoints but did not define the runtime, evidence graph, privacy lifecycle, tenant policy, query semantics, durable usage facts, recovery model, or executable release gates required for a world-class operator product. Staff-engineering audits further showed authorization fail-open, incomplete cross-store deletion, trace join gaps, and assembly correctness defects that would make a premature Track D UI dishonest. The redesign therefore places readiness and control-plane work before dashboard UI and treats each capability as a vertical slice with contract and assurance evidence.
Current fact versus proposed target
| Area | Current fact at the review baseline | Proposed target; not implementation evidence |
|---|---|---|
| Public web | web/ is the public documentation, roadmap, benchmark, and marketing site. | Keep authenticated operator routes out of the public documentation application. |
| Dashboard surfaces | services/dashboard/ is a small provisional 4.P.0 connection shell; the broader operator mock is a separate fixture-backed prototype. | Make services/console/ the canonical operator application and keep services/dashboard/ compatibility-only until a retirement gate passes. |
| D1 data path | The readiness review identified useful backend foundations but did not establish a completed D1 production integration. | D1 must use real context, Overview, platform-health, and SSE contracts with no fixture production path. |
| D2 evidence | Full composed Explore/Trace read contracts and all provenance joins were not verified as complete. | D2 starts metadata-first and expands only when provenance, freshness, completeness, redaction, and tenant-negative evidence gates pass. |
Canonical dependency model
Use these graph terms everywhere: P0, P1, D0, P6-bootstrap, P6-slice-certification, D1, D2, D3, D4, D5, D6, E1, E2, E3, and E4. The acyclic dependency model is:
P0 → P1
P0 + P1 → (D0 ∥ P6-bootstrap)
D0 + P0 + P1 + P6-bootstrap → D1
P2 + P6-slice-certification → D2
P3 + P6-slice-certification → D3
P3 + P6-slice-certification → D4
P1 + P2 + P4 + P6-slice-certification → D5
P4 + P6-slice-certification → D6
P6-bootstrap + D1 + D2 evidence → E1
resilience evidence → E2
E1 + E2 → E3
all P/D evidence + E1 + E2 + E3 + full F4 register disposition (including F4-035) → E4P6-bootstrap establishes the harness and manifest contract. P6-slice-certification is produced separately for each D slice after that slice's required P contract is available; it is not a claim that all assurance work is complete. D0 and P6-bootstrap may proceed in parallel only after P0 and P1 are available. E1 consumes P6-bootstrap plus D1/D2 evidence, E2 consumes resilience evidence, E3 follows E1 and E2, and E4 signs off all evidence and the full F4 register, including F4-035 surface separation.
The graph is a proposed sequencing contract, not a claim that any step is complete.
Findings and disposition
Product and control-plane findings
| ID | Finding | Severity | Disposition |
|---|---|---|---|
| F4-001 | The public web deployment is static documentation output and cannot provide cookie-backed dynamic dashboard behavior or a long-lived SSE endpoint by itself. | P0 | planned/specification in 4.P.0 with an explicit server-capable or SPA/API/SSE topology. |
| F4-002 | Dashboard identity, org switching, resource authorization, step-up assurance, and asynchronous-job scope were not a single enforceable contract. | P0 | mounted-but-provisional in 4.P.1: permission bits, kill switches, TOTP/step-up scaffolding, and fail-closed binding on the probe exist; not a single enforceable HTTP contract yet (require_operator_permission / RequireStepUpProbe unmounted; org switching deferred). Remainder: #846. |
| F4-003 | Aggregate trace rows cannot explain nested causality, checkpoint joins, candidate ranking, fallback, tool, or evaluation provenance. | P0 | mounted-but-provisional in 4.P.2: nested Postgres evidence runs/spans + outbox + CH companion projections; Trace Inspector UI still Track D. |
| F4-004 | ClickHouse TTL and browser confirmation are not coordinated deletion, legal hold, or data-subject erasure. | P0 | implemented-with-evidence in 4.P.3 for operator erasure: receipted multi-store saga + hold-gated CH mutations + tombstones/watermarks. residual-risk: MergeTree TTL may still age rows under an active hold (#853; not a deletion SLA). |
| F4-005 | The old cost milestone assumed durable usage and billing facts that were not defined at the required org/agent/provider/model scope. | P1 | planned/specification in 4.P.4 and 4.D.6 with immutable usage facts, rate-card versions, reconciliation, and enforcement. |
| F4-006 | Generic permission constants do not define raw-data, export, deletion, replay, policy-change, secret-use, or break-glass authorization. | P0 | mounted-but-provisional in 4.P.1 + 4.P.3: ADR-0009 bits + _STEP_UP_MASK + LegalHoldManage + capture modes landed; resource/field/action HTTP enforcement for raw/export/delete/replay/secret-use remains open (#846). |
| F4-007 | Realtime streams need durable positions, resume semantics, bounded buffers, dedupe, and partial-data states. | P1 | mounted-but-provisional in 4.P.2: evidence outbox provides durable aggregate_seq positions; operator SSE hub (4.P.0) remains the live transport. Full slow-client verification stays 4.E.2. |
| F4-008 | Prompts, outputs, tool arguments, links, HTML, and Markdown are untrusted data and must never become browser actions or unsanitized markup. | P0 | mounted-but-provisional in 4.P.3: capture modes default metadata_only, async redaction, privileged archive encryption. Safe-rendering / hostile-content DOM suite remains open for Track D (4.D.2–4.D.4). |
| F4-009 | The previous Trace Inspector did not separate score contribution, rank transition, resource cost, and selection outcome. | P1 | planned/specification in 4.D.2 and UI/UX §15 with three-layer progressive disclosure and exclusion groups. |
| F4-010 | Replay, counterfactuals, policy changes, and fallback are high-impact actions without a common preview, approval, ledger, and rollback contract. | P0 | planned/specification in 4.P.1, 4.P.2, and 4.D.5. |
| F4-011 | Backup/restore, RPO/RTO, rollout abort, image provenance, and admission checks were described but not required release evidence. | P0 | mounted-but-provisional in 4.P.5: Helm overlays, pgBackRest→MinIO scripts, Cosign image digests, Kyverno Enforce policy text, backup freshness alerts, and a real compose/podman restore-drill with tenant_isolation_post_restore: true (infra/scripts/platform/evidence/4p5-restore-drill/). Postgres rpo_pass stays false on pg_dump_fallback (not PITR). Staging Kyverno admit + pgBackRest+WAL RPO and chaos/load remain #869 / 4.E.2–4.E.3. |
| F4-012 | Existing dashboard E2E planning did not prove four roles, two tenants, accessibility, hostile-content rendering, or static-site/runtime separation. | P1 | planned/specification in 4.P.6 and 4.E.1. |
| F4-013 | Drift and directive management were combined without an explicit producer contract for fingerprints, baselines, evaluations, or alert lifecycle. | P1 | Keep directive/routing controls in 4.D.5; integrate drift only after Phase 4.5 producer contracts are published. |
| F4-014 | Documentation status did not expose the redesigned Phase 4 milestone graph or active work state. | P1 | planned/specification in the Phase 4 index, current-state page, tracks, goals, risks, and this register. |
| F4-035 | The repository has multiple operator-facing surfaces with different roles: public docs, a provisional compatibility shell, and a fixture-backed product prototype. Treating them as one production dashboard would create an unverified runtime and data path. | P0 | in-progress in 4.D.0: services/console is now the canonical implementation boundary with an explicit preview-only fixture gate; live backend/auth integration, deployment evidence, compatibility cutover, and the retirement gate remain open. |
Engineering blockers before Track D honesty
| ID | Finding | Severity | Disposition |
|---|---|---|---|
| F4-015 | When Postgres is unavailable, the proxy can construct a PassthroughRegistry that allows every model for every organization (fail-open authorization). | P0 | Implemented in merged PR #897 (df657fd); runtime evidence remains open: PassthroughRegistry and the operational bypass are removed, provider selection fails with 503 SERVICE_DEGRADED, and missing-dependency/no-provider tests exist. Keep the runtime gate open until canonical deployment evidence is attached. |
| F4-016 | Model-policy invalidation uses lossy Postgres→Redis Pub/Sub dual-write, allowing stale allow decisions after revoke. | P0 | implemented-with-evidence in 4.P.1: durable org_model_policy_meta.epoch bumped in the same txn as policy mutations; cache stores epoch; EpochPoller backstop. Negatives: epoch event validation + poller/unit coverage in packages/modelpolicy. |
| F4-017 | Auth GetProviderCredential can return plaintext secrets without a narrowly scoped write/service authorization. | P0 | implemented-with-evidence in 4.P.1: Get requires permissions.SecretUse after org match; Create/List/Delete stay on OrgSettingsWrite. Negative: TestUnit_GetProviderCredential_Paths denies OrgSettingsWrite-only callers. |
| F4-018 | BYO provider BaseURL SSRF boundary is not enforced at Auth and proxy hot path. | P0 | implemented-with-evidence in 4.P.1: shared packages/ssrf validates on Auth Create/Get and proxy pin before BaseURLOverride. Negatives: private/metadata literals blocked in packages/ssrf + TestUnit_ApplyCredentialOverride_BlocksPrivateBaseURL. |
| F4-019 | Organization deletion can report success after Postgres-only cleanup while ClickHouse and object-storage data remain. | P0 | implemented-with-evidence in 4.P.3: fail-closed IBEX_ORG_DELETION_DEPLOYED_STORES (unconfigured deployed stores → store_unreachable, never verified); receipts require verified/not_applicable; worker-integration proves PG+CH+Redis+MinIO absence. |
| F4-020 | API readiness is startup-only; deletion/maintenance tasks lack soft/hard time limits. | P1 | mounted-but-provisional in 4.P.3 + 4.P.5: org-deletion 300/600; billing rollup 60/120; reconcile 300/600; maintenance 300/600. API /ready dependency-aware (4.P.0). Scheduled retention/soft→hard purge still open. |
| F4-021 | Context-assembly RPC lacks durable request_id / trace_id propagation required to join Trace Inspector sections. | P0 | implemented-with-evidence in 4.P.2: Assemble request/response carry request_id/trace_id/span_id; proxy mapper no longer drops directive_version_id. |
| F4-022 | Checkpoint IDs are not returned from writes (CheckpointID nil), so checkpoint cannot be a stable bridge. | P0 | implemented-with-evidence in 4.P.2: AppendCheckpoint returns checkpoint UUID; post-response path sets TraceRecord.CheckpointID before CH emit. |
| F4-023 | AssemblyMetrics / stage timings are computed but not persisted on the durable trace read model. | P1 | implemented-with-evidence in 4.P.2: persisted on evidence_assembly_metrics (+ CH companion) and llm_traces.context_assembly_ms. |
| F4-024 | Applied session migrations store hashes, not conversation transcripts; no audited MinIO/archive write path for full payloads. | P0 | implemented-with-evidence in 4.P.3: packages/objectstore + worker objectstore_client AES-256-GCM archive Put/DeletePrefix; capture redaction can persist privileged payloads when policy allows. Live MinIO exercised in worker-integration multistore absence test. |
| F4-025 | Score contract mismatches UX: interim similarity/confidence mix must not be labeled as five-component composite; real weights and exclusion reasons are missing. | P1 | mounted-but-provisional in 4.P.2: versioned interim_v1 score payload + exclusion field persisted; full five-component weights remain deferred. |
| F4-026 | Per-request directive snapshot (immutable version/hash) is not persisted with the inference. | P1 | implemented-with-evidence in 4.P.2: evidence_directive_snapshots + CH directive_version_id dual-write. |
| F4-027 | Tool-call audit lacks sanitized args, idempotency key, and session/trace join fields. | P1 | mounted-but-provisional in 4.P.2: evidence_tool_audits schema + PersistRun support; MCP hot-path population deferred with degraded label. |
| F4-028 | Context budgeting omits tool schemas / formatter wrap from available-token accounting (usable budget must leave room for tools + per-memory wrap before format). | P1 | implemented-with-evidence in merged #873: BudgetCalculator subtracts tool-schema tokens and fixed formatter overhead; assembly uses wrap-aware per-memory costs (cost_by_id) then post-trims for separators. |
| F4-029 | Configured rank weights are validated but ignored at scoring time. | P1 | implemented-with-evidence in merged #873: Settings.rank_weights() is wired into rank_hydrated_hits and compute_hot_cache_score, with env-to-field and regression coverage. Residual: existing Redis hot-cache ZSET members keep write-time ordering until rewrite/eviction. |
| F4-030 | Multi-label half-life uses primary label only; packer repair can return lower-value sets. | P1 | implemented-with-evidence in merged #873: hydrate categories from memory_labels; packer local-improve recovers the documented counterexample; multi-label half-life tests cover read and hot-cache paths. |
| F4-031 | Embedding profile/version persistence and pre-score truncation ordering defects undermine retrieval provenance. | P1 | Fix memory/retrieval contracts referenced by 4.P.2 and 4.D.3. |
| F4-032 | No executable K8s/Helm/Kustomize overlays or proven backup/restore drill for operator topology. | P0 | mounted-but-provisional in 4.P.5: infra/helm/ibex-harness/ executable; restore-drill proven on live Postgres with RLS isolation evidence attached. Production/staging cluster restore + Kyverno admit remain #869 / 4.E.2. |
| F4-033 | SSE write deadlines, queue shedding, and provider connection caps remain incomplete for operator-grade realtime. | P1 | planned/specification under 4.C.5 release-evidence soak + 4.P.0 / 4.E.2 (retain 4.C.5 implementation; evidence still required). |
| F4-034 | Python CI / Semgrep intended as merge blockers are not fully protected as required branch checks. | P1 | Declared in merged PR #897 (df657fd); live protection remains open: Python and custom-rule Semgrep aggregate contexts, always-reporting gates, and source regression guards exist. Keep open until hosted checks pass and an authorized owner applies/verifies the exact contexts in live GitHub branch protection. Community rules remain advisory. |
| F4-036 | Proxy's environment loader and Config.ApplyDefaults() could default a missing IBEX_ENV to development, bypassing non-development Redis requirements and production-only config guards. | P0 | Implemented in merged PR #897 (df657fd); runtime configuration evidence remains open: explicit profile validation and local/staging/production Helm profile wiring exist. Keep the deployment gate open until staged values and Redis Secret health are verified in the canonical runtime environment. |
| F4-037 | Proxy's Redis-backed organization rate-limit middleware fails open when Redis operations fail, allowing paid-provider work without the configured shared limit; missing Redis also selected an always-allow Noop outside explicit production-profile checks. | P0 | Implemented in merged PR #897 (df657fd); runtime outage evidence remains open: non-development Redis is required, operation failures return 503 before downstream work, and local/tagged outage tests exist. Keep the runtime gate open until the canonical deployment demonstrates the outage behavior. |
Closure rule
A finding reaches implemented-with-evidence only when the referenced milestone has a merged implementation and reproducible verification artifact. A finding reaches residual-risk only with an accepted residual-risk record, owner, review date, and operational rollback or mitigation decision. A roadmap page alone is evidence of intent, not closure. Zero open P0 findings is mandatory for Phase 4 exit (4.E.4).
Follow-up (F4-028/029/030): #873 is merged. Record its reproducible verification artifacts for F4-028/029/030. If an artifact is missing, retain that finding at planned/specification, or use residual-risk only with an accepted residual-risk record.
Related pages
Edit on GitHub
Last updated on