M3)¶

The top-level map of how epistemic-graph scales from a Raspberry Pi to an HA cluster while staying a durable source of truth and responsive under a 24/7 ingestion firehose. This page ties the three waves together and links out to the deep references — it does not duplicate them.

North star¶

One engine that saturates the hardware it is on while staying responsive. Three properties, held at every scale:

Durable source of truth. A kill -9 never loses an acked write — redb is authoritative by default (CONCEPT:AU-KG.backend.backend-modes), commit-before-ack (CONCEPT:EG-KG.backend.authoritative-dispatch), read-through-safe eviction (CONCEPT:EG-KG.storage.read-through-seam-exercised).
Non-blocking under ingestion load. The __commons__ ingestion firehose must not serialize the box or starve interactive reads. This is the whole point of the write coalescer, the sharded writer, MVCC reads, and the reserved read lane.
Scales single-node → cluster. The same binary is an embedded library, a single durable server, or — with the opt-in cluster layer — a replicated multi-node cluster. Runs on Raspberry Pi 4+ up to a 64-core box.

The program is three waves:

Wave	Theme	What it buys
M1	Single-node durable throughput	Saturate one box's cores for durable writes; keep reads responsive.
M2	HA cluster	Replicate the authoritative store across nodes (openraft 0.10 multi-Raft).
M3	Horizontal / elastic scale	Distribute tenants across shards/nodes, reshard online, offload cold tenants.

flowchart LR
    M1["M1 — single-node durable throughput<br/>redb-authoritative · coalescer · group-commit · K-way writer · MVCC reads"]
    RESP["Responsiveness layer<br/>reserved read lane · pipelining · pooled conns · parallel fan-out"]
    M2["M2 — HA cluster<br/>openraft 0.10 multi-Raft · durable redb log"]
    M3["M3 — horizontal scale<br/>tenant catalog · online reshard · rebalancer · cold offload · BLOB stream"]

    M1 --> RESP
    M1 --> M2
    M1 --> M3
    M2 -.cross-node moves.-> M3

M1 — single-node durable throughput¶

The foundation: make one box a durable source of truth that saturates its cores for writes without serializing on a single lock or a single fsync. The canonical detailed description is the "Durability model" section of AGENTS.md; this is the spine.

Authoritative durability (the floor)¶

redb-authoritative by default (CONCEPT:AU-KG.backend.backend-modes) — a stock build with a persist dir is durable out of the box; the one-time .mp/.wal → redb migration runs on first authoritative boot.
Commit-before-ack (CONCEPT:EG-KG.backend.authoritative-dispatch) — a durable mutation is group-commit fsynced to redb before its Response is acked; a commit failure is an ERROR response, so an acked write is always on disk.
Read-through-safe eviction (CONCEPT:EG-KG.storage.read-through-seam-exercised) — the per-graph node cap keeps RAM bounded without data loss: an evicted node serves from redb on a RAM miss, and a node leaves RAM only after a redb read confirms it is on disk.
Backpressure, not drop — the writer's bounded channel blocks for capacity off-reactor instead of shedding a mutation.

Throughput: how one box stops serializing¶

Four layered optimizations turn the durable path from "one lock, one fsync, one core" into "K writers, batched fsyncs, parallel cores":

Per-graph write coalescer (CONCEPT:EG-KG.sharding.per-graph-write-coalescer) — N concurrent single-op writes to ONE hot graph batch onto a lazily-created per-graph writer and apply under one topo.write() per batch, collapsing N lock acquisitions into ⌈N/batch⌉. Default ON, auto-sized from cpu count. → write_coalescer.md.
Adaptive group-commit micro-linger (CONCEPT:EG-KG.backend.adaptive-linger-coalesce) — when the eg-redb-writer is about to commit a shallow batch it spends ONE bounded recv_timeout(linger) (default 1 ms) letting concurrent in-flight authoritative writers fold into the SAME fsync — the profiled write ceiling was ~1 op/fsync from serial awaits. Adaptive: a deep batch already coalesces, so it commits immediately (no added latency). Durability is unchanged.
O(1) audit-chain tail cache (CONCEPT:EG-KG.storage.embedded-store) — the EG-KG.sharding.row-level-security hash-chained AUDIT log appends in O(1) by caching the per-graph chain tail in-process, so a high-rate ingestion stream no longer pays a durable tail read on every audited write.
Sharded K-way durable writer (CONCEPT:EG-KG.backend.sharded-k-way-durable) — redb is single-writer-PER-FILE, so the writer shards by graph into K independent graph-<n>.redb files, each with its own writer thread / bounded channel / Pending. All of the above (micro-linger, audit cache, commit-before-ack, group-commit, backpressure) hold per shard, so K cores commit in parallel. A graph always routes to the same shard by FNV-1a(sanitized_name) % K; K = clamp(cpu/2, 1, 8). K=1 is byte-for-byte the old single-graph.redb path (and forced to 1 under an active Raft node). → engine.md § Sharded K-way durable writer.

Reads that never block on the writer¶

Snapshot/MVCC reads off the writer (CONCEPT:EG-KG.storage.snapshot-read-off-writer) — the point-read / read-through path serves an evicted node directly off a Database::begin_read() MVCC snapshot on the target shard, concurrently with the single writer. A read never routes through the writer thread and never forces a group-commit. Consistency: reads see the latest committed state per shard, and commit-before-ack guarantees any acked write is already committed (so visible to a later begin_read()).

Sizing the box automatically¶

Dynamic capacity auto-sizing + Pi-OOM cap (CONCEPT:AU-KG.backend.b-auto-size) — the same binary sizes its concurrency / buffer / per-graph-node-cap defaults from (cpu_count, total_RAM) at startup. max_in_flight becomes cpus*64 (256 on a Pi, 4096 on a 64-core box); the per-graph node cap becomes (ram/2)/2048 so a 1 GiB Pi caps a runaway graph instead of OOM-killing every tenant, while a 247 GiB box stays effectively unbounded. Safe because eviction is read-through (no data loss).

The M1 write path¶

flowchart LR
    P["concurrent producers<br/>(per-graph Tokio tasks)"]
    COAL["per-graph write coalescer<br/>(EG-KG.sharding.per-graph-write-coalescer) — ⌈N/batch⌉ topo.write()"]
    SHARD["shard_for(name) = FNV-1a(name) % K<br/>(EG-KG.backend.sharded-k-way-durable)"]
    W0["eg-redb-writer 0<br/>graph-0.redb"]
    Wk["eg-redb-writer K-1<br/>graph-(K-1).redb"]
    GC["group-commit fsync<br/>(EG-024 micro-linger folds awaiting writers)"]
    ACK["ack (durable) — commit-before-ack (KG-2.187)"]

    P --> COAL --> SHARD
    SHARD --> W0
    SHARD --> Wk
    W0 --> GC
    Wk --> GC
    GC --> ACK

The M1 read path (never the writer)¶

flowchart LR
    R["read / query"]
    LANE{"reserved read lane<br/>(EG-KG.coordination.reserved-read-lane) if writes saturate admission"}
    SNAP["MVCC snapshot:<br/>in-mem GraphCore snapshot (Cypher/SQL/GraphQL)<br/>OR redb begin_read() (read-through, EG-KG.storage.snapshot-read-off-writer)"]
    RESULT["result — never a write lock, never a group-commit"]

    R --> LANE --> SNAP --> RESULT

The responsiveness layer¶

Durable throughput is only half of "saturate the box while staying responsive." These keep interactive traffic alive while ingestion runs flat-out:

Reserved read-admission lane (CONCEPT:EG-KG.coordination.reserved-read-lane) — an interactive MCP read/query is never shed BUSY behind a write firehose. Under __commons__ saturation both the global max_in_flight pool and the per-graph cap fill; a write that loses is back-pressured (BUSY, retry), but a read falls back to a dedicated read_admission semaphore writes can never acquire, bypassing the per-graph cap. Only a genuine read flood that also fills the small reserved lane is shed. → reserved_read_lane.md (full page).
Pooled / multiplexed engine connections (CONCEPT:EG-KG.backend.multiplexed-connections, roadmap E) — client-side: the Python ConnectionPool / ShardRouter auto-size to the box (2*cpu clamped 8..64) and fan independent ops across N connections; the engine spawns one task per connection, so N connections = N parallel server tasks.
True single-connection request pipelining (CONCEPT:EG-KG.backend.framed-response) — server-side: the per-connection loop tokio::io::splits the stream and tokio::spawns a dispatch task per frame whose id-tagged Response is written back out of order through a single writer task — so many requests on ONE connection process concurrently. Composes with EG-037 (the pool multiplexes connections AND each connection multiplexes requests).
Parallel cross-shard read fan-out (CONCEPT:AU-KG.backend.roadmap-f-parallel-cross) — a cross-shard read (load_all/load_into) fans each shard's dump concurrently off its OWN begin_read() MVCC snapshot on the blocking pool (K reads on K cores), never touching a writer thread. → m3_resharding.md.

M2 — HA cluster (openraft 0.10 multi-Raft)¶

The opt-in cluster layer (cargo raft feature, cluster-only — the main default/full build links no openraft) runs the engine as a multi-node HA cluster replicating its authoritative redb state via openraft 0.10 (CONCEPT:AU-KG.backend.authority-has-already-acked — the v2 split-storage API + native graceful leader transfer). Off ⇒ the write path is byte-for-byte the single-node path.

Durable redb Raft log (CONCEPT:EG-KG.storage.one-fsync-covers-raft) — the Raft log + vote + applied state live in the SAME graph.redb as the graph data, so a log append and its mutation coalesce into one WriteTransaction / one fsync; a restarted node recovers its log tail locally. The separate raft.redb sidecar is gone.
Multi-Raft scaffold (CONCEPT:EG-KG.sharding.raft-resharding) — a MultiRaft manager holds N openraft groups keyed by GroupId, sharing ONE TCP listener per node (frames tagged + demuxed by group id) and ONE shared graph.redb; a GroupRouter maps graph_name → GroupId. Group = transaction boundary.
M2 hardening (CONCEPT:AU-KG.ontology.manage-arbitrary/266/267/268/271) — pooled per-peer connections, group-per-tenant-range routing ring, per-group snapshot scoping, multi-node membership join, leader balancing (now the native trigger().transfer_leader(target) handoff), and heartbeat coalescing — all done + lib-tested.

K=1 under Raft

Under an active Raft node the durable writer is forced to K=1 (one group = one serialized write path) — M2 is HA, not write-scaling. Multi-Raft sharding (many write groups) is a separate, off-by-default direction (EG-KG.sharding.raft-resharding/2.266).

Validate the cluster mechanism (formation / replication / failover / native transfer / durable log) on throwaway loopback nodes with scripts/validate-raft-cluster.sh. The DONE-vs-REMAINING handoff (incl. what still needs real multi-node hardware) is m2_raft_status.md; the deploy recipe is cluster_deployment.md.

M3 — horizontal / elastic scale¶

The "horizontal scale spine": distribute tenants across the K durable shards (and, behind M2, across nodes), reshard online, and offload cold tenants — so 100M graphs can share an engine without all being resident. Full handoff: m3_resharding.md.

Capability	Concept	One-liner
Tenant catalog routing-override seam	`EG-031`	Durable `graph → {shard, node}` map that overrides EG-KG.backend.sharded-k-way-durable hash routing per graph; an empty catalog is byte-for-byte FNV-1a.
Catalog auto-attach gate	`EG-KG.sharding.r5-feature`	`RedbBackend::open` attaches `catalog.redb` only under `EPISTEMIC_GRAPH_TENANT_CATALOG=1` or an existing populated catalog — default OFF, pure EG-026.
Offline K-shard migration tool	`EG-030`	OFFLINE `migrate-shards` CLI rewrites a store into a new K, routing every graph with the SAME FNV-1a; copies rows verbatim (encryption + audit chain survive).
Online single-node resharding	`EG-032`	Move ONE graph between shards while the engine RUNS, then flip the catalog route; a `routing_epoch` `RwLock` quiesces only the move.
Online-reshard snapshot+delta copy	`EG-KG.backend.flush-pending-first`	Shrinks the moved graph's write-pause from O(graph) to O(delta): bulk-copy off a snapshot with writes flowing, then re-export + delta-flip-purge under the quiesce.
Rebalancing planner	`EG-035`	PURE, deterministic greedy hottest→coldest planner emitting an ordered `Vec<ReshardMove>`; emits, never executes.
Rebalance plan execution	`EG-KG.backend.r3-plan-execution`	`rebalance_execute(plan)` applies the plan move-by-move via online resharding, one graph at a time.
Cold-tenant whole-graph offload	`EG-KG.sharding.eg-r6`	Hibernates graphs idle longer than a window (durably-gated, read-through-safe — no loss); `__commons__` never offloaded.
Cold-tenant `touch()` + sweep	`EG-KG.backend.r6-feature`	Wires the tracker into the live read/write path + an interval offload sweep (`EPISTEMIC_GRAPH_COLD_OFFLOAD_SECS`).
In-process BLOB streaming facade	`EG-KG.sharding.m3-r4`	Streams a multi-GB blob between a `Read`/`Write` and the content-addressed CAS without buffering the whole blob.
Catalog-driven resharding admin RPC	`EG-038`	The wire surface (`Reshard`/`Catalog*`/`RebalancePlan`/`RebalanceExecute`) that drives the M3 ops over the protocol.
Parallel cross-shard read fan-out	`AU-KG.backend.roadmap-f-parallel-cross`	Cross-shard reads fan concurrently off per-shard `begin_read()` snapshots (also part of the responsiveness layer above).

Still remaining: R2 (cross-node tenant distribution — needs M2), and the object-store arm of R6 (cold tenants colder than redb spilled to cold-tier-s3/blob-s3).

Tiers — what scales where¶

There is one main build plus the opt-in cluster layer (full map: tiers.md; feature flags in AGENTS.md). The scaling capabilities partition as:

Capability	main build (`default`/`full`)	`+ cluster`
redb-authoritative durability (M1 floor)	✅	✅
Write coalescer · K-way sharded writer · MVCC reads · group-commit linger	✅	✅
Reserved read lane (EG-KG.coordination.reserved-read-lane) · auto-sizing (AU-KG.backend.b-auto-size)	✅	✅
Tenant catalog · online reshard · rebalancer · cold offload (M3, `redb`)	✅	✅
BLOB CAS + streaming facade	✅	✅
DataFusion SQL / pg-wire (+ the whole wire family)	✅	✅
openraft multi-Raft replication (M2)	❌	✅
Cross-node resharding (M3 R2) · distributed Pregel · cross-shard 2PC	❌	✅

The invariant

The main build (default/full) links no openraft — HA replication and cross-node distribution force the opt-in cluster layer. Everything else, including DataFusion SQL and the M3 single-node resharding machinery, is in the one main build. It targets Raspberry Pi 4+.

Operating notes¶

Key environment knobs¶

Variable	Layer	What to tune
`EPISTEMIC_GRAPH_REDB_SHARDS`	M1	K independent writer files/threads. Default `clamp(cpu/2,1,8)`. K=1 on a Pi; raise toward cores on a many-core box. FIXED per persist-dir (changing it needs the `migrate-shards` tool). Forced to 1 under Raft.
`EPISTEMIC_GRAPH_REDB_GROUP_LINGER_US`	M1	Group-commit micro-linger (default 1000 µs; `0` = legacy commit-on-drain).
`EPISTEMIC_GRAPH_REDB_GROUP_SHALLOW`	M1	Shallow-batch op threshold the linger applies below (default 32).
`EPISTEMIC_GRAPH_REDB_FLUSH_THRESHOLD`	M1	Per-shard early-flush op threshold (auto ≈ half the WAL queue depth, clamped 256..16384).
`EPISTEMIC_GRAPH_MAX_INFLIGHT`	M1 / resp.	Global admission cap (default `cpus*64`, 256 on a Pi / 4096 on 64-core). Excess → `BUSY`.
`EPISTEMIC_GRAPH_READ_RESERVED`	resp.	Reserved read lane size (default `max_inflight/8` clamped 8..1024).
`EPISTEMIC_GRAPH_COLD_OFFLOAD_SECS`	M3	Idle-offload sweep window (`0`/absent = disabled).
`EPISTEMIC_GRAPH_TENANT_CATALOG`	M3	Attach the durable tenant catalog (default OFF = pure FNV-1a).
`EPISTEMIC_GRAPH_RAFT_NODE_ID` / `_PEERS` / `_BIND_ADDR`	M2	Activate the cluster (with `--features raft` + a persist dir).

What to tune for a Pi vs a 64-core box¶

Raspberry Pi 4+: leave REDB_SHARDS at the K=1 default (one core, one file — adding shards just adds threads it can't parallelize). The auto-sizer already caps max_in_flight at 256, the per-graph node cap at ~262 k resident nodes (1 GiB), and floors the reserved read lane at 8. Consider COLD_OFFLOAD_SECS to bound RAM across many tenants. No openraft, no DataFusion.
64-core box: let auto-sizing pick REDB_SHARDS toward 8 (or raise with the migration tool if you have many hot graphs across many cores), max_in_flight ≈ 4096, reserved read lane ≈ 512. This is where the K-way writer + parallel cross-shard fan-out actually saturate the cores.
HA cluster: build --features cluster, set the RAFT_* env, and remember the writer is K=1 per node under Raft — scale write throughput by adding groups (multi-Raft), not shards.

How to verify¶

Transport / admission + responsiveness: python3 scripts/bench_transport.py (measured baseline: AddNode p50 ≈ 0.187 ms over UDS) and the transport::tests reserved-lane suite (see reserved_read_lane.md).
Write coalescer: write_coalescer::tests::bench_inline_vs_coalesced (57.5× fewer lock acquisitions under a pipelined firehose — see write_coalescer.md).
Cluster: scripts/validate-raft-cluster.sh on loopback nodes (formation / replication / failover / native transfer / durable log).
Prometheus: watch epistemic_graph_read_reserved_admitted_total, epistemic_graph_busy_rejections_total, the per-graph epistemic_graph_write_batches_total / …_batched_ops_total, and graph_memory_bytes / budget_evictions_total on the /metrics listener.

AGENTS.md — canonical "Durability model" section (the source of truth for M1/M2 mechanics + the Environment Variables table).
engine.md — the master-of-all engine deep reference (C4 views, all modalities).
write_coalescer.md — CONCEPT:EG-KG.sharding.per-graph-write-coalescer in depth.
reserved_read_lane.md — CONCEPT:EG-KG.coordination.reserved-read-lane in depth.
m2_raft_status.md — M2 DONE-vs-REMAINING handoff.
m3_resharding.md — M3 DONE-vs-REMAINING handoff.
tiers.md — feature-composition map + prebuilt binary sizes.
concepts.md — the concept registry.