Skip to content

Tenant-Partitioned Engine Sharding

CONCEPT:AU-KG.sharding.tenant-partitioned-sharding-hrw (sharding) · CONCEPT:AU-OS.scaling.shard-topology-visibility-per (topology visibility)

Stage-2 scaling for the epistemic-graph compute tier: run N independent engine processes ("shards") and let every client route to the right one — no proxy hop, no coordinator, no engine changes.

The partition model in one line

tenant  →  named graph  →  HRW (rendezvous hash)  →  shard endpoint
  • The named graph is the partition unit. Each engine process keeps its own string-keyed named-graph registry; a graph lives wholly on exactly one shard, so single-graph operations never need cross-shard coordination.
  • Tenancy enters only by choosing the graph name. When a caller does not target an explicit graph and the ambient ActorContext (CONCEPT:AU-OS.identity.authenticated-identity-enforcement) carries a tenant, the default graph is mapped to tenant__<tenant>__<base> by tenant_graph_name() (knowledge_graph/core/shard_topology.py, also exported from agent_utilities.knowledge_graph and as KnowledgeGraph.tenant_graph()).
  • Shard choice is a pure function of the graph name. The sync client path (GraphComputeEngine) delegates to the exact HRW implementation in epistemic_graph.pool.ShardRouter, so sync and async callers can never disagree on placement.

Configuration

Flag (on AgentConfig) Default Meaning
GRAPH_SERVICE_ENDPOINTS unset Comma-separated or JSON list of authenticated coordinator/bootstrap contacts (unix:// / tcp:// / tls://). Unset or one entry = today's single-engine behaviour (zero-infra preserved); 2+ entries enable sharding. Contacts are not placed-group authority: authenticated ClusterMembers provides the current group endpoint, leader, epoch, and certificate metadata.
GRAPH_CLUSTER_ID unset Optional pinned sha256:<digest> cluster identity. When unset, the first authenticated ClusterMembers response binds the process-local cache; a restart must rediscover.
GRAPH_CLUSTER_DISCOVERY_MAX_AGE_S / GRAPH_CLUSTER_DISCOVERY_CLOCK_SKEW_S 30 / 5 Bounds for last-good member discovery and certificate validity. Expired, stale, or context-incompatible snapshots fail closed.
GRAPH_DRAIN_TIMEOUT_S 15 Bounded wait for active GraphOS/MCP calls during process or pod drain; timeout never claims continuity.
KG_DEFAULT_GRAPH __bus__ The default named graph; the ambient tenant maps onto tenant__<t>__<default> in sharded mode only.

Routing-key resolution (resolve_routing_graph):

  1. explicit, non-default graph name → used verbatim;
  2. ambient ActorContext tenant → tenant_graph_name(tenant, default);
  3. otherwise → the configured default graph.

Bootstrap/contact strings are hashed verbatim — configure every client with the identical list (order does not matter; HRW is order-independent) and with explicit schemes. For a placed group, the live client never substitutes a static GRAPH_RAFT_GROUP_ENDPOINTS entry or a route-provided endpoint hint: it must use the verified, freshness- and certificate-bounded ClusterMembers snapshot. The legacy map remains parseable only for migration and configuration audit.

Operational semantics

  • Autostart is local-only. Autostart is not an independent toggle — it is derived purely from endpoint topology (engine_resolver.setting_autostart) and may spawn an engine only for a local (unix://) endpoint. In sharded mode an unreachable remote (tcp://) shard is a fail-loud ConnectionError naming the shard, the graph it owns, and the remediation — the same hard-contract convention as the CONCEPT:AU-KG.backend.selectable-queue-backend task queue. Auto-starting a local stand-in will silently split that shard's graphs into invisible islands.
  • The flock host role is per-host. host_lock.py elects ONE daemon owner per host for the local engine; remote shards are reported by the status surfaces, never managed.
  • Auth is fleet-wide. All shards and all clients must share ONE GRAPH_SERVICE_AUTH_SECRET (CONCEPT:AU-OS.identity.authenticated-identity-enforcement). Set it explicitly in multi-host deployments — the auto-generated per-install secret only covers one host.

Placement catalog (DIST-P2-2b) — HRW is now the fallback, not the only authority

knowledge_graph/core/placement_catalog.resolve_placement asks the engine's own authoritative PlacementCatalog (epistemic-graph raft/placement.rs) for a graph/tenant's owning endpoint, caching the (endpoint, epoch) answer for a short TTL (PLACEMENT_CATALOG_TTL_S, default 5s) and re-resolving on a stale-epoch redirect.

This section is stale and the "graceful HRW fallback" it describes no longer matches the deployed engine — corrected 2026-07-31, see reports/deferred/lane-webui-dataplane.md (D-WD-1, D-WD-2). The engine does now expose a wire PlacementRoute RPC (proven live against the cluster), so the "no wire RPC yet, therefore always falls back" premise below is false. There is also no callable HRW-ring implementation to fall back to — shard_topology.shard_endpoint_for does not exist in this repository (repository grep, 2026-07-31); whatever module this once named has since been removed or renamed. The real, current failure mode for a non-cluster-admin principal is not a silent HRW fallback — it is ACCESS_DENIED from the engine's own require_admin_capability gate (PlacementRoute is authz_action = "admin:cluster-read"), surfaced to the caller as PlacementAuthorityError. PLACEMENT_CATALOG_ENABLED (core/config.py) remains unread by any code path (D-WD-2) — it is reserved as the natural switch for making placement resolution in security.request_identity._mint_graph_session conditional (D-WD-1), not yet wired. Treat the paragraphs below as the original design intent, not today's behavior.

Rebalancing (out of scope — the honest caveat)

HRW keeps key movement minimal when a shard is added or removed (~1/N of graphs change owner), but no data moves automatically. A graph whose HRW winner changed re-creates empty on its new shard until the operator migrate it manually with the existing snapshot tooling:

  1. quiesce writers for that graph;
  2. export from the old shard (lifecycle.to_msgpack / GraphComputeEngine.to_msgpack(), or copy its --persist-dir checkpoint);
  3. import on the new shard (from_msgpack) after the endpoint list changed everywhere;
  4. remove the stale copy from the old shard.

Durable mirrors (e.g. pg-age) are unaffected — they are not partitioned by this mechanism.

Topology visibility (CONCEPT:AU-OS.scaling.shard-topology-visibility-per)

  • shard_topology_status() → shard mode, per-endpoint transport-level reachability probe, locality, and circuit-breaker state. Surfaced on:
  • the unified daemon status (unified_daemon_status()["shards"], i.e. GET /daemon/status and python -m agent_utilities.gateway.daemon --status);
  • the gateway dashboard route GET /daemon/shards;
  • graph-os GET /health (cheap config-only summary: shard_mode, shard_count — no probe on the liveness path).
  • Prometheus (AU-OS.observability.no-op-without-metrics registry, agent_utilities/observability/gateway_metrics.py):
  • agent_utilities_engine_shard_up{endpoint} — 1/0, refreshed on every real client connect and by the status probe;
  • agent_utilities_engine_shard_requests_total{endpoint,outcome} — the engine-call outcomes (ok | connection_error | error | short_circuited) split per shard;
  • the existing agent_utilities_gateway_engine_breaker_state{endpoint} is already per-endpoint, so each shard gets its own circuit breaker for free.
  • Each engine process can also expose its own native metrics with --metrics-addr (epistemic_graph_* series, one scrape target per shard).

Worked example — 3 shards on one host

See docker/engine-shards.compose.yml for the runnable compose file (3 engines, distinct ports + persist dirs + metrics listeners, one shared secret), or by hand:

export GRAPH_SERVICE_AUTH_SECRET="$(openssl rand -hex 32)"
for i in 1 2 3; do
  epistemic-graph-server \
    --tcp-addr "127.0.0.1:910${i}" \
    --persist-dir "/var/lib/epistemic-graph/shard-${i}" \
    --metrics-addr "127.0.0.1:911${i}" &
done

# Every agent-utilities client / gateway / ingest worker:
export GRAPH_SERVICE_ENDPOINTS="tcp://127.0.0.1:9101,tcp://127.0.0.1:9102,tcp://127.0.0.1:9103"

Multi-host is the same picture with one engine (or a small number of, on big hosts) per machine and hostnames in the endpoint list. Capacity planning for shard counts lives in docs/scaling/capacity_model.md (RESIDENTS_PER_ENGINE_SHARD).

Part of the Knuckles-Team agent ecosystem. Source repository · Documentation home