Skip to content

Metrics Reference

Consolidated catalogue of every Prometheus metric series the agent-utilities Python tier emits. All series below are registered in ONE module — agent_utilities/observability/gateway_metrics.py (CONCEPT:AU-OS.observability.no-op-without-metrics) — and incremented by the subsystem modules listed per table. Names, types, labels, and meanings are verified against that module's registrations and docstrings.

How to scrape

  • The gateway (and the agent-webui backend — both mount register_graph_routes) exposes GET /metrics in Prometheus exposition format. The endpoint is exempt from the identity middleware and rate limiting (scrapers cannot mint JWTs).
  • prometheus_client is the optional metrics extra. Without it every series degrades to a shared no-op (recording costs ~nothing) and /metrics returns a self-describing placeholder with HTTP 200.
  • The whole middleware + endpoint is toggled by GATEWAY_METRICS (default on); see ../architecture/configuration.md.
  • Metrics live in the default per-process registry. With GATEWAY_WORKERS>1 (pre-forked workers on one listen socket) a scrape samples ONE worker — aggregate across replicas in Prometheus or run one worker per container (see ../architecture/gateway_scaling.md).
  • Request-duration histogram buckets: 5ms to 60s (0.005 ... 30, 60).
  • Worked scrape/dashboard setup: ../examples/observability.md.

Cardinality discipline: the route label is always a route TEMPLATE (/api/graph/{name}), never a raw path — unmatched requests (404/405) collapse into the single unmatched bucket so internet scanners cannot mint series. endpoint cardinality is bounded by the configured GRAPH_SERVICE_ENDPOINTS list; server by the children declared in mcp_config.json.

Gateway HTTP (CONCEPT:AU-OS.observability.no-op-without-metrics)

Recorded by GatewayMetricsMiddleware (pure ASGI, mounted outermost by gateway/graph_api.py::register_graph_routes so 401/429 rejections are counted too; /metrics itself is not instrumented).

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_gateway_requests_total Counter route, method, status Gateway HTTP requests by route template, method, and status code observability/gateway_metrics.py (middleware) AU-OS.observability.no-op-without-metrics
agent_utilities_gateway_request_duration_seconds Histogram route Request duration by route template observability/gateway_metrics.py (middleware) AU-OS.observability.no-op-without-metrics
agent_utilities_gateway_in_flight_requests Gauge — Requests currently being handled by this process observability/gateway_metrics.py (middleware) AU-OS.observability.no-op-without-metrics
agent_utilities_gateway_rate_limited_total Counter tenant Requests rejected (429) by the per-tenant token-bucket rate limiter gateway/rate_limit.py AU-OS.observability.no-op-without-metrics

Engine client & circuit breaker (CONCEPT:AU-OS.observability.no-op-without-metrics)

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_gateway_engine_requests_total Counter op, outcome epistemic-graph engine client calls by operation and outcome (ok | connection_error | error | short_circuited) knowledge_graph/core/engine_breaker.py AU-OS.observability.no-op-without-metrics
agent_utilities_gateway_engine_breaker_state Gauge endpoint Engine circuit-breaker state per endpoint (0=closed, 1=half-open, 2=open) knowledge_graph/core/engine_breaker.py AU-OS.observability.no-op-without-metrics

Engine shard topology (CONCEPT:AU-KG.sharding.tenant-partitioned-sharding-hrw / AU-OS.scaling.shard-topology-visibility-per)

One series per configured GRAPH_SERVICE_ENDPOINTS entry. The reachability gauge is refreshed on every real client connect attempt and by the daemon's shard_topology_status probe (surfaced through gateway/api.py); the per-shard counter splits the engine-call outcomes so a hot or failing shard is visible at a glance. The existing breaker gauge above is already per-endpoint, so each shard gets its own circuit breaker.

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_engine_shard_up Gauge endpoint Per-shard engine reachability (1=reachable, 0=unreachable) knowledge_graph/core/shard_topology.py AU-KG.sharding.tenant-partitioned-sharding-hrw / AU-OS.scaling.shard-topology-visibility-per
agent_utilities_engine_shard_requests_total Counter endpoint, outcome Engine client calls per shard endpoint and outcome (ok | connection_error | error | short_circuited) knowledge_graph/core/engine_breaker.py AU-KG.sharding.tenant-partitioned-sharding-hrw / AU-OS.scaling.shard-topology-visibility-per

KG ingest queue backpressure (CONCEPT:AU-KG.ingest.decoupled-kg-ingest-consumer)

Sampled by the KG maintenance scheduler on the leader host. Depth is uniform across queue backends (sqlite/postgres = row count, kafka = kg-ingest consumer-group lag); the lag series exists separately so Kafka dashboards and alerts read naturally. These two series also feed the autoscaler's zero-infra LocalMetricsProvider signals (queue_depth, consumer_lag) when SCALING_PROMETHEUS_URL is unset (CONCEPT:AU-OS.scaling.reactive-replica-autoscaling, orchestration/scaling_signals.py). Each value gauge has a paired _observed_at Unix-time gauge with the same labels. The telemetry writer publishes each pair through one locked snapshot authority. LocalMetricsProvider accepts only exact value/companion label pairs and rejects missing, mismatched, duplicate, stale, or high-cardinality pairs; Prometheus scrape timestamps and local read time are never substituted.

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_kg_ingest_queue_depth Gauge backend Pending KG ingest tasks in the selected durable task queue knowledge_graph/core/engine_tasks.py AU-KG.ingest.decoupled-kg-ingest-consumer
agent_utilities_kg_ingest_queue_depth_observed_at Gauge backend Unix observation time for the paired queue-depth gauge knowledge_graph/core/engine_tasks.py AU-KG.ingest.decoupled-kg-ingest-consumer
agent_utilities_kg_ingest_consumer_lag Gauge topic, group Total kg-ingest consumer-group lag (unconsumed messages) per topic knowledge_graph/core/engine_tasks.py AU-KG.ingest.decoupled-kg-ingest-consumer
agent_utilities_kg_ingest_consumer_lag_observed_at Gauge topic, group Unix observation time for the paired consumer-lag gauge knowledge_graph/core/engine_tasks.py AU-KG.ingest.decoupled-kg-ingest-consumer

Embedded MCP fleet child resilience (CONCEPT:AU-ECO.mcp.profile-differences-from-client)

One series per aggregated child server (~50, bounded by mcp_config.json). GraphOS emits these from its embedded fleet gateway; like every series here, they degrade to no-ops when the metrics extra is absent.

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_mcp_child_calls_total Counter server, outcome Multiplexer tool calls per child and outcome (ok | error | transport_error | timeout | busy | unavailable | short_circuited) mcp/child_resilience.py AU-ECO.mcp.profile-differences-from-client
agent_utilities_mcp_child_breaker_state Gauge server Per-child circuit-breaker state (0=closed, 1=half-open, 2=open) mcp/child_resilience.py AU-ECO.mcp.profile-differences-from-client
agent_utilities_mcp_child_restarts_total Counter server Automatic restarts of crashed child servers mcp/child_resilience.py AU-ECO.mcp.profile-differences-from-client
agent_utilities_mcp_child_queue_depth Gauge server Tool calls queued behind a child's concurrency limit right now mcp/child_resilience.py AU-ECO.mcp.profile-differences-from-client

Queue-driven agent dispatch (CONCEPT:AU-ORCH.dispatch.queue-agent-dispatch)

Sampled by the dispatch workers on their fleet-registry heartbeat tick. Depth is uniform across queue transports (kafka = agent-dispatch consumer-group lag, postgres/sqlite = row count).

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_dispatch_queue_depth Gauge backend Unclaimed dispatched agent turns in the agent_turns queue orchestration/agent_dispatch_worker.py ORCH-1.45
agent_utilities_dispatch_turns_total Counter outcome Dispatched agent turns processed by this worker process (completed | failed | skipped | expired) orchestration/agent_dispatch_worker.py ORCH-1.45
agent_utilities_dispatch_workers Gauge — Live agent-dispatch workers (fresh heartbeats in the fleet registry) orchestration/agent_dispatch_worker.py ORCH-1.45

Agent communication bus (CONCEPT:AU-ECO.bus.operator-view-agentbus)

The agent-to-agent bus (AgentBus, AU-ECO.bus.agentbus-federated-agent-agent). Participant gauges are sampled on the status/health read; message and dispatch counters and the send-duration histogram are emitted on the send/dispatch path. Labels are small enums, so cardinality stays flat.

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_bus_participants Gauge status Registered participants by computed presence (online | offline) messaging/bus.py AU-ECO.bus.operator-view-agentbus
agent_utilities_bus_messages_total Counter kind, outcome Bus messages by kind (direct | topic) and outcome (delivered | denied | no_recipient) messaging/bus.py AU-ECO.bus.operator-view-agentbus
agent_utilities_bus_send_seconds Histogram — Server-side latency of one AgentBus.send (gate + per-recipient durable write) messaging/bus.py AU-ECO.bus.operator-view-agentbus
agent_utilities_bus_dispatch_total Counter outcome Message→fleet-work dispatches (submitted | denied | failed) messaging/bus.py AU-ECO.bus.operator-view-agentbus

Load-test these with scripts/bench_bus.py against a live hub; the modeled expectation (participants/hub, msgs/s/connection) is in docs/scaling/capacity_model.py (bus_plan_for). The Grafana dashboard is agent-bus.json (generated by scripts/gen_grafana_dashboards.py).

ContextCompiler / Epistemic RAG (CONCEPT:AU-KG.retrieval.context-compiler / CONCEPT:AU-KG.retrieval.context-compiler-kv-seam)

The policy-aware context compiler (knowledge_graph/retrieval/context_compiler.py) is the mandatory evidence-assembly boundary every Pydantic AI agent goes through (core/contextual_model.py::create_context_agent/wrap_model_with_context) — this is the delegated-run (execute_agent/execute_workflow) default context path (W3.7), on by default, with the MODEL_CONTEXT_COMPILER_ENABLED deployment-level escape hatch as the only opt-out (see contextual_model._context_compiler_enabled). ITEMS/TOKENS are recorded on every compile() call; KV_CACHE only fires once a Seam-6 KV backend is configured (contextual_model.set_context_compiler_cache, opt-in — an external network-dependent performance layer, not part of the mandatory path). TTFT is recorded from TWO call sites, disambiguated by the path label so the interactive and batch latency populations are never averaged together: contextual_model._record_delegated_run_ttft (every governed execute_agent/ execute_workflow model call) and context_compiler_serving._record_ttft (bundle_chat_completion/bundle_async_chat_completion, the background enrichment/extraction call sites).

Name Type Labels Meaning Emitted by (module) Since (concept id)
agent_utilities_context_compiler_items_total Counter outcome Candidate items per compile() call by outcome (selected | dropped_policy | dropped_redundant | dropped_budget) knowledge_graph/retrieval/context_compiler.py AU-KG.retrieval.context-compiler
agent_utilities_context_compiler_tokens Histogram kind Per-compile() token counts (in = MMR-selected pool before budget fit, selected = tokens kept); selection efficiency = selected/in knowledge_graph/retrieval/context_compiler.py AU-KG.retrieval.context-compiler
agent_utilities_context_compiler_kv_cache_requests_total Counter outcome Seam-6 bundle-level KV-cache lookups (compile(kv_backend=...)) by outcome (hit | miss) — hit-rate = hit/(hit+miss) knowledge_graph/retrieval/context_compiler.py AU-KG.retrieval.context-compiler-kv-seam
agent_utilities_context_compiler_ttft_seconds Histogram kv_cache_hit, path Wall-clock latency of a governed model call using a compiled bundle — a time-to-first-token PROXY, split by KV-cache-hit and by call-site population (path="delegated_run" | path="bundle_chat_completion") core/contextual_model.py, knowledge_graph/retrieval/context_compiler_serving.py AU-KG.retrieval.context-compiler-kv-seam

Fleet autoscaler (CONCEPT:AU-OS.scaling.reactive-replica-autoscaling)

The autoscaler registers no metric series of its own. It consumes immutable, identity-bound ScalingSignalSample values: either this process's own gauges via the zero-infra LocalMetricsProvider (queue_depth → agent_utilities_kg_ingest_queue_depth, consumer_lag → agent_utilities_kg_ingest_consumer_lag), or bounded instant Prometheus HTTP queries (sum(...) over the same fleet-total series) when SCALING_PROMETHEUS_URL is set. Signal names are built-in or explicitly deployment-allowlisted; raw caller PromQL is not an autoscaler input. Each autoscaler tick uses one bulk provider call; Prometheus uses one client only for a thread-safe transport (otherwise one client per worker), schedules all allowlisted queries in bounded rolling waves, and caps unique queries, concurrency, and the overall deadline. Timed-out work retains its bounded in-flight slot until it finishes. The fixed four-worker pool uses daemon threads, so even a broken transport that ignores cancellation cannot prevent process shutdown. Fleet-total multi-series results are summed; per-replica multi-series results are averaged. Samples carry immutable unit and tenant/fleet scope metadata. Stale, replayed, malformed, cross-service, cross-unit/scope, and over-cardinality samples are treated as no data. The queue-depth and consumer-lag built-ins bind exactly to the kg-ingest-worker scale unit; other services require an explicit deployment definition. See orchestration/scaling_signals.py and orchestration/fleet_autoscaler.py.

Rust engine series (epistemic_graph_*)

The epistemic-graph engine exposes its own native epistemic_graph_* Prometheus series — these live in the engine repository, not in agent-utilities, and their exact names are defined by the engine's own documentation. What this repo wires and documents:

Series count

25 Python-tier series documented above: 4 gateway HTTP, 2 engine client/breaker, 2 shard topology, 2 ingest queue, 4 MCP child, 3 dispatch, 4 agent bus, 4 ContextCompiler — all registered in observability/gateway_metrics.py and verified against code in this pass. gateway_metrics.py registers additional series not yet catalogued here (KVCACHE_CLIENT_REQUESTS, LANE_IN_FLIGHT/LANE_QUEUE_DEPTH, MCP_TOOL_*, SCHEDULED_JOB_*, SKILL_CALLS, TOOL_CALLS, DB_CALLS) — a pre-existing documentation gap, out of scope for this pass and not tracked anywhere else.

Part of the Knuckles-Team agent ecosystem. Source repository · Documentation home