Milestone 3.5.E.4 — MCP Observability and Rate Limiting

Status: Completed Goal: Track E — MCP Tool Surface
Phase: 3.5 — Extraction & Context Assembly
Estimated effort: 1 day
Track: Track E — MCP Tool Surface
Depends on: Milestone 3.5.E.3 (record_feedback tool)

Planning baseline (sketch deviations)

Shipped vs sketch: Go packages/ratelimit / packages/circuitbreaker are not reused from Python — mcp-memory implements a calendar-minute Redis Lua twin (INCR + EXPIRE-on-create, 90s TTL) keyed ratelimit:<org_id>:mcp:<unix_minute> (independent of proxy :rpm:). Redis errors fail open. Over-limit rejects raise MCPServiceError(code="rate_limited") → FastMCP CallToolResult(isError=true) — not HTTP 429 (matches existing in-tool PermissionDenied precedent). Default RPM IBEX_MCP_RATE_LIMIT_RPM=120 with optional IBEX_MCP_RATE_LIMIT_ORG_OVERRIDES. Auth breaker polish is hand-rolled (N=5 consecutive AuthUnavailableError, 30s cooldown, half-open single probe) in front of the existing timeout fail-closed path. ibex.mcp_tool_calls already exists (000002, 90-day TTL); this milestone does not CREATE TABLE again — it gap-fills auth-reject audit rows (tool_name=auth) and internal_error codes for unexpected exceptions.

Why This Milestone Exists

An MCP server exposed to external agent frameworks is a new, untrusted-by-default entry point — it needs the same operational rigor as the HTTP proxy.

Requirements

Every tool call traced to ClickHouse mcp_tool_calls table (org_id, agent_id, tool_name, latency_ms, success, error_code) — table created in 2.5.G6.M1 / ADR-0050; this milestone must not CREATE TABLE again
Rate limiting uses a Python Redis calendar-minute limiter keyed ratelimit:{org_id}:mcp:{unix_minute} — independent budget from chat-completions (not Go packages/ratelimit)
Circuit breaker polish: AuthService gRPC unreachable → short-circuit after N failures (fail-closed timeout already shipped in G6)

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

MCP server surface (services/mcp-memory/)
Redis (shared cluster, separate key namespace)

Success signals

Outcome-oriented signals that the milestone is in good shape. Exact filenames, package layouts, and commands may differ from any sketches above.

mcp_tool_calls wiring verified (existing 000002 + 90-day TTL); auth-reject + internal_error audit gap-fill
Rate limit test: over-budget search_memory / tool invoke → isError with error_code=rate_limited (not HTTP 429); key uses :mcp: not :rpm:
Auth breaker: N consecutive AuthUnavailable → open short-circuit; cooldown → half-open; success closes
Repo guards / CI checks still pass

Prerequisites

Milestone 3.5.E.3 (record_feedback tool)

Milestone 3.5.E.4 — MCP Observability and Rate LimitingCopy Anchor Link

Why This Milestone ExistsCopy Anchor Link

RequirementsCopy Anchor Link

Orientation (indicative)Copy Anchor Link