Deployment

v1-single-node profile

The supported single-node Docker Compose profile — one published HTTPS port, pinned dependencies, safe bootstrap, and recorded readiness.
Implemented Ledger class: shipped-local
Verified 2026-11-07Locally started and recorded on one host; not a hosted, HA, or multi-user claim

The recorded start is 29-v1-single-node-start-record-2026-11-07.md.

v1-single-node is the only deployment profile a v1.0 claim may cite (ADR-0097). It runs every required service as a container in one Compose project on one host. Kubernetes, Helm, and any HA topology are outside the profile.

Start it

make v1-secrets   # generates .env (0600) and file secrets; never prints a value
make v1-up        # builds, migrates, provisions the capability logins, bootstraps, starts, waits for the edge
make v1-ps        # service status

make v1-secrets writes the four capability-login passwords as well (IBEX_RUNTIME_DB_PASSWORD, IBEX_AUTH_DB_PASSWORD, IBEX_WORKER_DB_PASSWORD, IBEX_BOOTSTRAP_DB_PASSWORD). Leave them alone: the one-shot provision service is their only writer, and it refuses to start when one is missing rather than defaulting to a shared value.

make v1-up prints the edge URL. The bootstrap writes the admin PAT once to infra/compose/v1-single-node/bootstrap/bootstrap.pat (mode 0600, directory 0700) through a host bind mount, so the credential survives the one-shot container and never reaches the container logs or the process output:

make v1-bootstrap-pat   # prints the PAT; store it, then delete the file

Only the Argon2id hash is stored in Postgres, so that file is the only copy. Move the value into your secret manager and remove the file. IBEX_BOOTSTRAP_PAT_FILE is required for a first install: the command refuses to start rather than print the credential to stdout, where a job runner would retain it. On a rootful runtime the file is owned by root; make v1-bootstrap-pat then reads it through the auth image, which runs as root inside its own namespace.

make v1-up waits for every required service and covers TEI's two-hour first-start window (the model download). It prints a progress line every two minutes; set IBEX_V1_HEALTH_ATTEMPTS to a smaller number to fail fast on a host that should already be warm.

What runs

ServiceImageInternal portReadiness
Edgecaddy:2.11.9-alpine (pinned digest)8443/_edge/health
Proxybuilt from services/proxy8080/ready
Authbuilt from services/auth8081, gRPC 9091/ready
Management APIbuilt from services/api8010/ready
Memorybuilt from services/memory8005/ready
Embedderbuilt from services/embedder8004/ready
TEI (BAAI/bge-m3)ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.4 (pinned digest)80/health
Workerbuilt from services/worker8007/ready
Context assemblybuilt from services/contextloopback gRPC 9092grpc.health.v1
Consolebuilt from services/console3000/ready
Postgres + pgvectorpgvector/pgvector:0.8.6-pg16 (pinned digest)5432pg_isready
Redisredis:7.4.2 (pinned digest)6379redis-cli ping
One-shot migratebuilt from infra/migrations—exits 0
One-shot provisionsame image as migrate—exits 0
One-shot bootstrapsame image as Auth—exits 0

Every long-running service declares a healthcheck. The one-shots run in order: migrate (schema), provision (capability-login credentials), bootstrap (identity). embedder, edge and console gate on migrate; bootstrap, auth, proxy, api, memory and worker gate on provision, which itself gates on migrate; and context gates on the proxy, memory, bootstrap and the edge. A failed migration or a failed provisioning run stops the profile instead of producing a partial start. The exact digests, environment matrix, and dependency graph live in infra/compose/v1-single-node/compose.yml; they are the source of truth for this page.

Network surface

Only the edge port is published: ${IBEX_EDGE_PORT:-443}:8443. Datastores, AuthService gRPC, context, memory, worker, and MCP ports are reachable only by service name on the internal bridge. The edge answers 404 for /health, /ready, and /metrics; the proxy's internal auth-probe route (/v1/internal/...) is not routable through the edge.

Rootless Podman and Docker cannot bind port 443 without additional privileges. Set IBEX_EDGE_PORT=8443 in infra/compose/v1-single-node/.env and use https://localhost:8443.

Path ownership is explicit per ADR-0097 §4:

PathOwner
/v1/chat/completions, /v1/responses, /v1/sessions/*/terminate, /v1/orgs/*/auth-probeProxy
/v1/operator/*, /v1/organizations/*, /v1/tokens/*, /v1/users/*, /v1/agents/*, /v1/tenant/*Management API
Everything elseConsole

Transport

  • Browser → edge: TLS with the per-install edge leaf, issued by the profile's own CA. Trust that CA on the client before opening the console.
  • Edge → proxy/API/console: these four containers share one network namespace, so the credential-bearing hops are loopback plaintext and never cross the Compose bridge (ADR-0097 §4). The edge cannot be replaced while the co-located containers exist, so replacing it is a full restart: make v1-down && make v1-up. make v1-up performs that restart itself when the certificate fingerprint changes. Rotate certificates in infra/compose/v1-single-node/certs/: secrets/ holds the container copies, and make v1-secrets overwrites it from certs/ on every run, so a key replaced only under secrets/ is silently reverted and the edge keeps serving the old leaf.
  • Console BFF → edge: TLS with the same CA through NODE_EXTRA_CA_CERTS; the browser never sees the upstream origin.
  • Proxy/API → AuthService: gRPC with TLS 1.2+ and the per-install CA (IBEX_AUTH_GRPC_CA_FILE).
  • Policy snapshots: AuthService signs them with the generated policy key; the proxy verifies them with policy_public_keys.json (key id v1). Staging and production refuse to mount protected routes without a verifier.
  • Proxy ↔ context: the context container shares the same network namespace and binds 127.0.0.1:9092 only. The profile bounds assembly at 2 s on both sides (proxy client and context retrieval) because CPU TEI query embedding cannot fit the 45 ms development default; the chat path remains fail-open and bounded.
  • Internal HTTP calls: service tokens on the Compose bridge; no published port. The shared bearer tokens and the Redis password are a documented limitation of the host-local bridge, not a public interface.

Multi-line PEM keys cannot live in a Compose .env. The profile uses file-backed variables instead: JWT_PRIVATE_KEY_FILE, IBEX_AUTH_POLICY_SIGNING_KEY_FILE, DASHBOARD_JWT_PUBLIC_KEYS_FILE, and optionally IBEX_PROXY_POLICY_PUBLIC_KEYS_FILE. Inline values still win when both forms are set.

Posture

The tracked infra/compose/v1-single-node/.env.example starts in IBEX_ENV=staging with IBEX_LLM_MODE=mock, so a clean host needs no external provider. The mock model is a deterministic fixture for exercising transport, authentication, and context assembly — it is not a language model.

The supported deployment posture is:

IBEX_ENV=production
IBEX_LLM_MODE=live
OPENAI_API_KEY=<operator-supplied>

Never run mock mode in production, and never commit the filled .env. make v1-secrets creates infra/compose/v1-single-node/.env with mode 0600; the secret copies under infra/compose/v1-single-node/secrets/ are 0444 inside a 0700 directory so the non-root containers can read them.

Identity bootstrap

The one-shot bootstrap service creates, in one transaction:

  • the organization (IBEX_BOOTSTRAP_ORG_NAME / IBEX_BOOTSTRAP_ORG_SLUG),
  • the owner user and the default agent,
  • an allow-all model policy for that organization,
  • an admin PAT (Admin | SecretUse | OperatorMetadataRead), and
  • a scoped memory service token (MemoryRead | MemoryWrite) used by context and worker.

It is idempotent by organization slug: a second run creates no organization and never rotates the PAT. It does reconcile the scoped memory token, so a changed IBEX_MEMORY_API_TOKEN takes effect on the next start; a revoked token is never reactivated. The committed development seed token is never used.

Capability logins

The profile starts three one-shots in order: migrate, then provision, then bootstrap. The migrations create the capability logins ibex_runtime, ibex_auth, ibex_worker and ibex_bootstrap inert — no password, CONNECTION LIMIT 0 — and provision runs migrate provision to give each one a SCRAM verifier and lift that limit, in a single transaction. It refuses a secret that is missing, not printable ASCII without spaces, repeated, or equal to the owner credential, and it checks the postcondition before committing: each login is a non-superuser NOBYPASSRLS, NOINHERIT, NO CREATEDB, NO CREATEROLE login with no connection limit and no expired validity, reaches only the roles it needs, and reaches no privileged identity. A stray GRANT fails the start instead of widening a login.

Every application service connects on its capability login. The profile's six application DSNs name the login directly, and each long-running service asserts it at startup — refusing to run on the owner login or a superuser:

ServiceLoginDSN in compose.yml
bootstrapibex_bootstrap:268
authibex_auth:306
proxyibex_runtime:349
apiibex_runtime:405
memoryibex_runtime:452
workeribex_worker:548
migrate, provisionowner ibex:209, :239

migrate and provision keep the owner credential because provision writes each capability role's password and the migrations create cluster-global roles. Row-level security therefore applies to every application statement.

This was live-proven on 2026-11-15: all 13 health-checked containers healthy, no application connection is ibex, an unpinned cross-tenant read on ibex_runtime raises SQLSTATE 42501, and a route-level cross-tenant request returns 403. That is a live cross-tenant read plus the integration suites — not a full acceptance suite, and not a two-tenant browser journey. make repo-guards also runs infra/scripts/check-tenant-pin.py, which fails any tracked Go or Python file that writes app.current_org_id without pinning a tenant role first.

If provision exits non-zero, Compose refuses to start the services that depend on it. Postgres, Redis, the edge, the embedder, the migration one-shot and the console come up; auth, proxy, api, memory, worker and context do not, so the console has no upstream and no request is answered:

make v1-ps          # provision exited non-zero; the datastores are healthy and nothing else started
make v1-provision   # rebuilds and re-runs the one-shot; no teardown
make v1-up          # starts the services Compose withheld and waits for health

v1-provision repairs the credential write in place; it does not remove the running datastores. v1-up is idempotent, so it starts the dependent services after the one-shot succeeds without disturbing the containers that are already healthy.

Verify

# Paste the URL printed by `make v1-up`. The host port comes from
# IBEX_EDGE_PORT (443 by default), not from the container's internal 8443.
EDGE_URL=https://localhost
ca=infra/compose/v1-single-node/secrets/ca.crt
curl -fsS --cacert "$ca" "$EDGE_URL/_edge/health"                                 # ok
curl -sS --cacert "$ca" -o /dev/null -w '%{http_code}\n' "$EDGE_URL/health"       # 404
curl -sS --cacert "$ca" -o /dev/null -w '%{http_code}\n' "$EDGE_URL/v1/internal/auth-probe" # 404
make v1-ps                                                                        # every required service healthy

Trust the profile CA on the client before opening the console; the commands above verify the edge certificate instead of disabling verification.

Then authenticate a chat request with the bootstrap PAT and confirm the round trip through the proxy, context assembly, and the provider. A live production start must serve at least one authenticated chat request before any deployment claim is recorded.

Upgrade and rollback

Upgrade is a stop-the-stack operation. The IBEX images are built locally and their tags are overwritten on every make v1-up, so record the image IDs — not tags — before anything else.

  1. Record the deployed images while the profile is running:
    # Podman is the reference runtime; on a Docker host `docker image inspect
    # --format '{{.Id}}'` prints the same immutable ID.
    for ref in ibex-v1-auth:local ibex-v1-single-node-proxy ibex-v1-single-node-api \
               ibex-v1-single-node-memory ibex-v1-single-node-embedder ibex-v1-single-node-worker \
               ibex-v1-single-node-context ibex-v1-single-node-console ibex-v1-single-node-migrate \
               ibex-v1-single-node-provision; do
      printf '%s %s\n' "$ref" "$(podman image inspect --format '{{.Id}}' "$ref")"
    done | tee /tmp/ibex-v1-images.txt
    
  2. Flush and drain extraction work. A turn lives in the proxy's Redis buffer until its session ends, and an empty queue does not mean a task is finished. With the profile still running:
    a. Ask clients to stop sending new chats, then terminate active sessions from the console (or wait for them to go idle) so each turn's extraction enqueue is sent while the queue and worker are alive. A failed enqueue is logged by the proxy as extraction enqueue failed; buffer retained; check for that line before stopping the proxy. b. Stop the proxy so no new turn can be accepted: infra/scripts/dev-tool.sh v1 stop proxy. c. Wait until the queue is empty and nothing is running, reserved, or scheduled for retry:
    REDIS_PASSWORD="$(sed -n 's/^REDIS_PASSWORD=//p' infra/compose/v1-single-node/.env)"
    podman exec -e REDISCLI_AUTH="$REDIS_PASSWORD" ibex-v1-single-node-redis-1 redis-cli -n 1 LLEN extraction
    podman exec ibex-v1-single-node-worker-1 celery -A app.celery_app:celery_app inspect active
    podman exec ibex-v1-single-node-worker-1 celery -A app.celery_app:celery_app inspect reserved
    podman exec ibex-v1-single-node-worker-1 celery -A app.celery_app:celery_app inspect scheduled
    

    Repeat until LLEN returns 0 and the three inspect calls list no tasks. A queued or buffered task's turn text is not reconstructable from Postgres, and the snapshot cannot recover it.
    Every enqueue uses priority 0, so one LLEN covers the queue. The worker runs with task_ignore_result=True, so Celery stores no task outcome: the queue length and the three inspect outputs are the completion signal, and the worker log is the only place a failed task becomes visible. This profile has no dead-letter queue.
  3. make v1-down (volumes preserved).
  4. Check out the target release (git checkout <tag>). The application and migration images are built from this checkout, so building from an older checkout starts old code under the new tags. Pinned dependency images (Postgres, Redis, TEI, Caddy) come from the Compose file; the IBEX images are built locally.
  5. Take a database snapshot.
  6. Run make v1-secrets before provisioning. An upgrade from the previous release has a .env without the four capability-login passwords (IBEX_RUNTIME_DB_PASSWORD, IBEX_AUTH_DB_PASSWORD, IBEX_WORKER_DB_PASSWORD, IBEX_BOOTSTRAP_DB_PASSWORD). The script appends the missing keys, keeps every existing value, and never prints a value; skipping it makes provision stop with IBEX_RUNTIME_DB_PASSWORD is required; run make v1-secrets.
  7. make v1-migrate, then make v1-provision (each rebuilds its image from this checkout and rewrites the capability verifiers from the current .env).
  8. make v1-up (rebuilds the application images, then waits for every service).

Before recording a deployment claim, rehearse both directions from a real snapshot of the previous release (the ADR names v0.1.9): run the upgrade, then run the rollback, and record both — the commands, the image IDs, and the row counts after the restore. A rollback that has never been executed is not evidence.

Rollback is a stop-the-stack operation and must not rebuild: a rebuild would replace the images you are trying to recover. Restore the snapshot, reset the transient Redis state (the extraction queue on DB 1, the result backend on DB 3, and the caches on DB 0/2 — recreate the redis-data volume or flush those databases), retag each recorded image ID onto its Compose name, then start without building:

set -e
while read -r ref id; do
  podman image inspect "$id" >/dev/null            # fail if the recorded image is gone
  podman tag "$id" "$ref"                          # fail if the tag cannot be set
done < /tmp/ibex-v1-images.txt
# On a Docker host, both steps are Docker's: `docker image inspect "$id" >/dev/null`
# fails the same way when the recorded image is gone, and `docker tag "$id" "$ref"`
# sets the same tag.
infra/scripts/dev-tool.sh v1 up --no-build         # IBEX_V1_SKIP_BUILD=1 has the same effect,
                                                   # and Compose receives `up --no-build`

The loop stops on the first failure: continuing after a failed tag would leave the upgraded tag in place and up could start the newer image against the restored snapshot.

Migrations are forward-only; starting an older image against a migrated schema is not a supported rollback. The database restore restores the token and identity state; Redis holds only transient queues, caches, and rate-limit counters. It also reverts the capability-login verifiers, which live in the database (pg_authid): the recreated one-shots rewrite them from the current .env, so a credential rotated after the snapshot must still be in .env when the stack starts.

Because the restore moves the database back in time, record and reapply anything that happened after the snapshot:

  • token and PAT revocations (revoked_at, is_revoked) and user suspensions, otherwise a revoked credential works again after the restore;
  • extraction work accepted after the snapshot, which the Redis reset discards and which must be re-submitted;
  • accepted organization-deletion requests and their outcomes (org_deletion_jobs), reconciled against the restored database: a request recorded succeeded is replayed, a hold_blocked request stays blocked (clearing its hold does not resume it), and a request that never left pending is submitted again;
  • legal-hold creations and clearances, re-applied before any forget or deletion replay runs. A restore from a snapshot that predates a hold would otherwise drop it silently and allow the destruction the hold exists to prevent;
  • completed forgets and organization deletions, replayed from the record kept outside the database. Re-apply the markers, erase the restored rows (see the next section), re-scrub the idempotency-ledger responses, and verify the receipts before reopening traffic. If the replay is incomplete, the rollback is incomplete.

Keep the window between snapshot and rollback short, and keep the revocation, deletion and hold records outside the database.

Traffic stays fenced for the whole rollback, including when the stack is started with v1 up --no-build: draining the queue (LLEN → 0) and the inspect calls prove the queue was empty at that moment, not that the restored data is deletion-complete. Reopen traffic only after the replay above is verified.

Data lifecycle on this profile

  • Forget erases content. POST /v1/memories/{id}/forget overwrites the memory's text, replaces its hash, zeroes its metadata and PII flags, deletes its labels, feedback, relationships and conflict escalations, and scrubs the finalized idempotent create response that stored the body. The row remains as a deletion marker, and a forgotten lifecycle event records the revision. Replaying the create's idempotency key afterwards returns 409 IDEMPOTENCY_KEY_ERASED, never the erased body; a create still finalizing when the forget commits stores the same marker instead of the content.
  • A legal hold blocks destruction. While ibex_core.legal_holds has an uncleared row for the organization, forget returns 409 LEGAL_HOLD_ACTIVE and organization deletion ends hold_blocked. Clearing the hold re-enables forget, but it does not resume a blocked deletion: that job stays hold_blocked, the organization is already cancelled, and a new deletion request for it is rejected. This is the recognised ground to refuse erasure when a legal obligation to preserve applies.
  • Organization deletion purges PostgreSQL and Redis in this profile (IBEX_ORG_DELETION_DEPLOYED_STORES=postgres,redis) and reports per-store receipts; a store declared deployed but unreachable fails the job rather than reporting success. ClickHouse and object storage are not part of this profile, so their purge paths are not claimed here. The ClickHouse tenant-table purge set exists (P1-20) but is provisional: only llm_traces is covered by a local absence test, so the other tables — usage_facts included — are implemented, not verified. The PostgreSQL billing tables are retained.
  • "Zero rows" excludes receipts and billing by design. org_deletion_jobs, deletion receipts, memory_lifecycle_outbox, ibex_billing.* and failed_tasks are retained; the organization row is marked cancelled with deleted_at. Do not read the deletion as "every row is gone". ClickHouse usage_facts is in the provisional purge set; the Postgres billing tables are retained.
  • Backups are not erased by forget. A restore moves the database back in time, so erasure is a recovery-wave obligation, not a side effect of forgetting:
    1. Record deletion events outside the database — forgets (memory, organization, revision), the scoped idempotency key of every erased create (organization_id, agent_id, dedupe_key, request_hash) and completed organization deletions with their per-store receipts. A snapshot that predates them does not contain them. Capture is crash-safe by reconciliation, not by ordering: every committed forget writes a forgotten row to ibex_core.memory_lifecycle_outbox in its own transaction, so after the fact compare that table and org_deletion_jobs against the external record. If the external record is missing an event the database has, or its completeness cannot be proven, treat the restore as unsafe and keep traffic fenced.
    2. After a restore, re-apply every deletion marker, then erase the restored row itself — overwrite content with [forgotten], replace content_hash with a new random value, zero content_tokens, reset metadata to {}, clear the PII flags, clear superseded_by/merged_into, clear retrieval_count/last_retrieved_at, and null embedding/embedding_model/embedding_dim — delete the restored child rows (labels, feedback, relationships, conflict escalations) and re-scrub the idempotency-ledger responses. A marker that only sets tombstoned leaves the content, hash, embedding and metadata recoverable in the restored row. If the ledger has no row for an erased key (it was purged, or the snapshot predates the row), insert a terminal marker so a retry cannot re-create the memory: INSERT INTO ibex_core.idempotency_ledger (organization_id, agent_id, dedupe_key, request_hash, state, status_code, response_payload) VALUES (..., 'finalized', 201, convert_to('{"erased": true}', 'UTF8')) ON CONFLICT (organization_id, agent_id, dedupe_key) DO UPDATE SET state = 'finalized', status_code = 201, response_payload = EXCLUDED.response_payload. A retry then returns 409 IDEMPOTENCY_KEY_ERASED instead of re-creating the memory.
    3. Verify before serving traffic: SELECT count(*) FROM ibex_core.memories WHERE tombstoned AND (content <> '[forgotten]' OR content_tokens <> 0 OR metadata <> '{}'::jsonb OR embedding IS NOT NULL OR pii_detected OR pii_redacted OR superseded_by IS NOT NULL OR merged_into IS NOT NULL) must return 0, every child-row count must be 0, every erased key must replay as 409 IDEMPOTENCY_KEY_ERASED, and the per-store receipts must match the external record. A restore with pending deletions, or one whose record cannot be reconciled, stays fenced. A Redis cache eviction that fails is retried only by the cache TTL, so a copy can survive there for IBEX_MEMORY_CACHE_TTL_SECONDS.
  • Database role. Memory CRUD, vector operations and the idempotency ledger run every transaction as ibex_app so row-level security applies; the ledger's queries are additionally scoped by organization_id. Nothing in the memory service runs with the service-account bypass. The login in IBEX_API_DATABASE_URL / the memory DSN must be a member of ibex_app; the profile's ibex role is granted it by migration 000004. A deployment that uses a different login must grant membership, or memory reads fail closed.

Exclusions and limits

  • No Kubernetes, Helm, or HA topology, and no hosted or measured p99 SLO.
  • No multi-user authentication: the profile bootstraps one owner and one agent.
  • No ClickHouse or MinIO/S3 projection, reranker, or observability stack; those stay optional and default off.
  • TEI runs on CPU. First start downloads roughly 2 GB of model weights into the tei-cache volume; the healthcheck allows for it. Expect CPU-bound embedding latency, not a GPU SLO.
  • The first start is gated on TEI and can take tens of minutes on a slow network: the TEI health window is two hours, and the embedder, memory, and worker wait behind it. The tei-cache volume keeps a partial download across restarts, so re-running make v1-up resumes rather than restarts the fetch. A host with a warm tei-cache skips the download entirely; that is the recommended pre-seed for an offline or bandwidth-limited install.
  • The reference profile is sized for a 16 GB host. TEI uses about 8 GB RSS with the 2048-token batch; the application plane is about 3 GB. On a smaller host, lower --max-batch-tokens and --max-concurrent-requests in infra/compose/v1-single-node/compose.yml, or give the host more memory. --max-batch-tokens also bounds the longest input TEI will accept: with the default --auto-truncate, TEI derives its per-request input limit from this value, and the embedder sends truncate: false, so a memory longer than the limit fails to embed instead of being shortened. Raise the value toward the model's context limit if long memories must be indexed.
  • The bootstrap PAT does not expire. Retrieve it once with make v1-bootstrap-pat, store it in a secret manager, and delete infra/compose/v1-single-node/bootstrap/bootstrap.pat afterwards. The command prints the credential to your terminal: do not run it from CI or any wrapper that retains stdout. Rotation is a revoke-and-reissue operation through the API, not a bootstrap rerun.
  • The generated memory service token is reconciled by a later bootstrap run when it changes. Stop the profile before changing IBEX_MEMORY_API_TOKEN, then run make v1-up: bootstrap rewrites the hashed row and recreates the context and worker containers that hold the old value. Editing .env alone leaves those two callers authenticating with the previous token until they restart.
  • One agent only. The memory service token is bound to the bootstrapped agent, so context retrieval for a subsequently added agent returns AGENT_NOT_AUTHORIZED and assembles without that agent's memories. Multi-agent memory authorisation needs a per-agent token design; it is not part of this profile.
  • Internal shared-token HTTP and the Redis password travel on the host-local Compose bridge. ADR-0097 records this as a documented limitation, not a public interface; the credential-bearing browser hops stay on loopback.
  • The profile is evidenced on one host with one tenant. Cross-tenant acceptance and recovery drills are later gates, and this page makes no claim about them.