Workers
The worker is a Python Celery process with four logical queues. It is not a single monolith task: each family has a different durability boundary.
Queues and order
celery -A app.celery_app:celery_app worker \
-Q extraction,embedding,maintenance,mcp_audit \
--loglevel=info
Cross-queue order is the -Q order. Intra-extraction priority uses Celery Redis priority steps 0–9. RabbitMQ x-max-priority is not used.
Task families
| Family | Responsibility | Status |
|---|---|---|
| Extraction | Read completed session turns, call an extraction provider, write memories | Implemented; live vLLM manual |
| Embedding | Async embedding jobs | Implemented |
| Organization deletion | Orchestrate tenant deletion and receipts | Implemented; certification open |
| Billing | Reconcile usage and roll up budgets | Implemented; hosted evidence open |
| Maintenance | Beat sweep and housekeeping | No-op/deferred |
| MCP audit | Fan out tool-call audit events | Implemented; sink durability varies |
Retry and dead letters
IbexTask owns the retry policy. There is no per-task on_failure override; lifecycle signals in app/observability.py handle failures.
Exhausted retries are persisted to ibex_core.failed_tasks and increment ibex_worker_task_dead_letter_total{task_name}. The alert rule is IBEXWorkerTaskDeadLettered.
A dead-letter row is evidence that retries were exhausted. It is not automatic remediation. Operators must inspect the task, decide whether replay is safe, and verify the downstream effect.
Enqueue admission
POST :8007/internal/extraction/enqueue accepts extraction work only when IBEX_WORKER_ENQUEUE_API_TOKEN is set. The endpoint requires that static bearer plus an Idempotency-Key of at most 256 characters.
Admission behavior:
- The normalized payload, including every turn, is hashed.
- The tenant-scoped PostgreSQL idempotency ledger claims the key before Celery dispatch.
- Replay returns the original task ID.
- A changed payload returns
409. - An active claim returns
409withRetry-After. - Dispatch failure releases the claim.
- If dispatch succeeds but finalization does not commit, the endpoint returns
503; retries may see409until the 120-second lease expires. - Without PostgreSQL, the endpoint fails closed with
503.
GET :8007/health is unauthenticated liveness only. GET :8007/ready checks the token, PostgreSQL ledger, and Redis broker.
Configuration highlights
| Variable | Default | Why it matters |
|---|---|---|
IBEX_WORKER_BROKER_URL | derived | Full broker override |
IBEX_WORKER_RESULT_BACKEND | derived | Full result-backend override |
REDIS_DB_QUEUE | 1 | Keeps queue keys out of cache DB |
REDIS_DB_RESULTS | 3 | Result TTL storage |
IBEX_WORKER_ENQUEUE_API_TOKEN | unset | Missing means the enqueue server does not start |
IBEX_WORKER_METRICS_PORT | 8006 | Prometheus exposition |
IBEX_WORKER_ENQUEUE_PORT | 8007 | Enqueue and readiness HTTP |
IBEX_WORKER_EXTRACTION_PROVIDER | openai | openai or vllm |
IBEX_WORKER_MEMORY_BASE_URL | unset | Memory service origin |
IBEX_WORKER_MEMORY_API_TOKEN | unset | Bearer with memory:write |
OTEL_SAMPLE_RATIO | 0.01 | Trace sampling |
Run locally
make compose-dev-up
make db-migrate
cd services/worker && uv sync --extra dev
export POSTGRES_DSN='postgres://ibex:[email protected]:5432/ibex?sslmode=disable'
celery -A app.celery_app:celery_app worker \
-Q extraction,embedding,maintenance,mcp_audit \
--loglevel=info
Use make worker-beat-dev for beat and make worker-ping for a broker health check.
Limits
- Queue registration is not a production-readiness claim.
- Result TTL and
task_ignore_resultdefaults matter for retry visibility. - The maintenance sweep is intentionally a no-op.
- There is no GPU CI runner; mock transports do not prove live vLLM behavior.
- Exactly-once projection into PostgreSQL, ClickHouse, and caches is not claimed.