ADR-0062: Worker task observability and dead-letter handlingCopy Anchor Link

• Status: Accepted
• Date: 2026-09-02
• Authors: IBEX Harness team
• Milestone: 3.5.A.2 Task observability and dead-letter handling

ContextCopy Anchor Link

Milestone 3.5.A.1 shipped the Celery worker skeleton with retry policy on IbexTask but no visibility into terminal failures. Without dead-letter handling, a systematically-failing task (misconfigured upstream, bad credentials) accumulates silently in Redis unacked state.

Requirements:

1. Every task execution emits an OTel span compatible with Go services (ADR-0019).
2. After retries are exhausted, persist failure context durably and increment a low-cardinality Prometheus counter.
3. Page operators when dead-letter rate is non-zero.

Options consideredCopy Anchor Link

1) Dead-letter destinationCopy Anchor Link

1. Redis / Celery native DLQ — requires a consumer and admin UI to be useful.
2. Flower as source of truth — operational convenience only; not a production dependency.
3. Postgres ibex_core.failed_tasks (chosen) — SQL-queryable immediately; survives Flower downtime; aligns with Phase 4 operator dashboard.

2) Span wrapping strategyCopy Anchor Link

1. Decorate every task function with @traced_task manually — error-prone as task count grows.
2. Override IbexTask.__call__ to call run_task_in_span (chosen) — guarantees coverage for all tasks using the base class.

3) RLS on failed_tasksCopy Anchor Link

1. Enable RLS with org filter — blocks cross-org forensics during incidents.
2. No RLS (chosen) — follows ibex_audit.audit_log pattern; inserts use app.is_service_account = true; reads restricted at application layer.

DecisionCopy Anchor Link

OpenTelemetry (first Python bootstrap)Copy Anchor Link

• Module: services/worker/app/telemetry.py
• Tracer name: ibex-worker
• Resource attributes mirror ADR-0019 (service.name, service.version, deployment.environment)
• Sampling: ParentBased(TraceIDRatioBased(OTEL_SAMPLE_RATIO)), default 0.01
• Exporter: OTLP gRPC when OTEL_EXPORTER_OTLP_ENDPOINT set; no-op provider otherwise (CI/unit tests)
• Init: Celery worker_ready signal (once per child process)

Per-task spans (run_task_in_span):

• Attributes: worker.task, celery.task_id, ibex.org_id / ibex.agent_id / ibex.session_id when present in kwargs
• Never log or span memory content
• On exception: record_exception, StatusCode.ERROR, re-raise

Dead-letter handlerCopy Anchor Link

Celery task_failure signal in app/observability.py:

Retry guard (critical):

if request is not None and request.retries < sender.max_retries:
    return  # intermediate attempt — autoretry follows

On terminal failure:

1. Increment ibex_worker_task_dead_letter_total{task_name} when a new failed_tasks row is inserted
2. Increment ibex_worker_task_dead_letter_persist_failed_total{task_name} when persistence is unavailable or errors
3. Persist via _run_on_worker_loop(insert_failed_task(...)) — uses the per-child event loop from worker_process_init (_worker_loop.run_until_complete); asyncio.run() is fallback only when no worker loop exists (tests / edge cases). SQLAlchemyError / asyncpg.PostgresError caught and logged.
4. Does not suppress the original task exception

Tenant context: org_id is required for tenant-scoped tasks; ibex.worker.maintenance.* tasks may persist with NULL org_id (system/maintenance failures). When a non-maintenance task exhausts retries without a valid org_id, persistence is skipped (no row inserted), ibex_worker_task_dead_letter_persist_failed_total increments, and an ERROR log (dead_letter_skipped_missing_org_id) is emitted — fail-closed, not a silent drop.

Idempotency: UNIQUE(task_id) on failed_tasks; IntegrityError → no-op.

PrometheusCopy Anchor Link

• HTTP /metrics via prometheus_client.start_http_server on IBEX_WORKER_METRICS_PORT (default 8006)
• Gauge ibex_process_up = 1 when metrics server started
• Alert IBEXWorkerTaskDeadLettered: rate(ibex_worker_task_dead_letter_total[5m]) > 0 for 1m, severity page

SchemaCopy Anchor Link

Migration 000021_create_failed_tasks — see DATABASE_SCHEMA.md § FAILED TASKS.

ConsequencesCopy Anchor Link

• Worker integration CI requires Postgres + db-migrate (mirrors memory-integration pattern).
• Dead-letter persistence reuses the per-child Celery event loop; asyncio.run() is fallback only — not a new loop per failure.
• Flower remains optional for live debugging; Postgres is the source of truth.
• Prefork Prometheus multi-process metrics (PROMETHEUS_MULTIPROC_DIR + MultiProcessCollector) are wired in code but lack a full scrape-after-fork integration test in CI — tracked in GitHub #675 (prefork Prometheus multi-process metrics validation).

ReferencesCopy Anchor Link

• ADR-0019: OpenTelemetry provider configuration
• ADR-0021: Prometheus metric catalog
• Migration: infra/migrations/postgres/000021_create_failed_tasks.up.sql
• Alert rule: infra/monitoring/prometheus/rules/ibex-worker.yml