Capacity Model (Plan 07: Path to Scale)

AU evidence level: The arithmetic implementation is IMPLEMENTED and its reference cases are UNIT-PROVEN when the focused model tests pass. The one-million row is DESIGNED/MODELED; it is not LAB-PROVEN, LIVE, or 1M-CERTIFIED. W3 manifests are reference/staged inputs until the release gate and an operator records deployment evidence. See the AU scale claim register.

Status (SCALE-P2-1): 1M is now a DEFINED workload contract, measured by a harness — not a linear-arithmetic claim. This page used to say "the 100M target follows as a measured projection" and stop there; Codex's SCALE-P2-1 review correctly called that out — a modeled shard/worker/node COUNT is not a demonstrated CAPACITY. The linear arithmetic below (capacity_model.py) is still here and still useful as a first-order infrastructure-sizing tool (how multiple PG shards/L0 shards/nodes to provision), but it is no longer what "1M" MEANS. What "1M residents, sustained" means now is defined precisely by workload_contract.yml + workload_contract.py — registered agents, concurrent sessions/turns, turns/tool-calls/graph-mutations/messages/tokens per second, tenant count + skew (incl. one elephant tenant), per-agent working-set/history/media footprint, interactive/background mix, availability + RPO/RTO, and p50/p95/p99/p99.9 SLO targets for queue/query/write/end-to-end latency — and it is GENERATED and MEASURED against those SLOs by scripts/scale/loadgen.py (a real driver, not more arithmetic), asserted by the soak/chaos harness in tests/scale/soak/. See "What is CI-measured vs hardware-pending" below for the honest current status: the CI-runnable subset actually runs and asserts invariants today; the real-scale/real-duration/ real-multi-node scenarios are documented and skip-marked, not faked.

The measured anchors below (transport latency, per-shard write throughput, per-agent working-set) still stand and now ALSO anchor multiple workload_contract.yml fields directly (see that file's # anchor: comments) so the contract and this model cannot silently drift apart — tests/scale/test_workload_contract.py cross-checks them.

The loader is an exact authority boundary: the population must be exactly 1,000,000, all rate/byte/availability values must be finite and non-negative, rates must agree with concurrency and duration, p99 byte bounds must be no smaller than their averages, and RTO must cover RPO. A loaded contract carries a content digest. Certification evidence must bind that digest to immutable release, deployment-topology, and workload-image digests and declare exactly one execution mode (live or mock). Live evidence without a live authority, or evidence that leaves mock/live authority ambiguous, is rejected; a mock run is never promoted to a live capacity claim. The service/loadgen integration still needs to supply those runtime digests and live-authority observation before a deployment can claim the full-scale result.

The measured anchor

The model imports one benchmark anchor from the engine project (run captured 2026-06-01, Linux x86-64, single client connection, in-memory graph, length-prefixed MessagePack over UDS):

Operation	ops	p50	p99
AddNode	3000	0.187 ms	0.223 ms

AddNode p50 ≈ 0.19 ms ⇒ ~5,000 sequential ops/sec on a single connection. This imported anchor is the only empirical input to this AU model; it is not evidence that the AU fleet is live at that rate. Throughput above one connection comes from connection pooling (pool.py) and shard fan-out (ShardRouter); the server sheds excess concurrent load with a BUSY response (EPISTEMIC_GRAPH_MAX_INFLIGHT, default 1024) rather than queueing unbounded.

Three-axis framing

Scaling is not a single dimension. This repository size three independent axes and take the max of the resulting infrastructure:

Axis	Driver	Knob in core/config.py
Active concurrency	agents running right now	worker_pool_size × node count; queue-driven dispatch is the implemented scale-out path for this axis — agent_dispatch_backend=queue + N agent-dispatch-worker hosts (see architecture/agent_dispatch.md, CONCEPT:AU-ORCH.dispatch.queue-agent-dispatch). Deployment/live status is separate evidence.
Resident population	total agents whose state must persist	graph_service_endpoints (PG/L0 shard fan-out — the L0 side is the implemented tenant-partitioned engine-sharding path, see architecture/engine_sharding.md, CONCEPT:AU-KG.sharding.tenant-partitioned-sharding-hrw). Deployment/live status is separate evidence.
Event throughput	graph events/sec driving fan-out	kafka_bootstrap_servers partitions

A deployment can be huge on one axis and tiny on another (e.g. 1M dormant residents with 2% active). Sizing each axis separately avoids over- or under-provisioning.

Per-unit planning constants (MODELED)

These live as named constants in capacity_model.py so they are testable and adjustable in one place:

Constant	Value	Meaning
RESIDENTS_PER_PG_SHARD	250,000	residents per durable Postgres shard
RESIDENTS_PER_ENGINE_SHARD	50,000	residents per hot in-memory engine shard (one GRAPH_SERVICE_ENDPOINTS entry; routed per named graph by HRW)
ACTIVE_AGENTS_PER_WORKER	25	concurrently active agents one worker multiplexes
WORKERS_PER_NODE	8	workers per node (= worker_pool_size default)
OPS_PER_SEC_PER_KAFKA_PARTITION	5,000	= the measured single-connection drain rate
EVENTS_PER_ACTIVE_AGENT_PER_SEC	2.0	modeled graph events per active agent
MIN_KAFKA_PARTITIONS	3	floor for ordering/parallelism headroom

OPS_PER_SEC_PER_KAFKA_PARTITION is the one constant tied to the measured anchor — one consumer connection drains ~5,000 ops/sec, so this repository size one partition per connection. The rest are conservative planning round numbers, not measurements.

The arithmetic (2% active fraction)

Active agents = ceil(residents × 0.02). Then:

• PG shards = ceil(residents / 250,000)
• L0 shards = ceil(residents / 50,000)
• Workers = ceil(active / 25)
• Nodes = ceil(workers / 8)
• Events/s = active × 2.0
• Kafka parts = max(3, ceil(events_per_sec / 5,000))

1,000 residents (MODELED, trivially within a single dev box)

• active = ceil(1000 × 0.02) = 20
• PG shards = ceil(1000/250000) = 1
• L0 shards = ceil(1000/50000) = 1
• workers = ceil(20/25) = 1, nodes = ceil(1/8) = 1
• events/s = 20 × 2 = 40, kafka = max(3, ceil(40/5000)) = 3

100,000 residents (MODELED)

• active = ceil(100000 × 0.02) = 2,000
• PG shards = ceil(100000/250000) = 1
• L0 shards = ceil(100000/50000) = 2
• workers = ceil(2000/25) = 80, nodes = ceil(80/8) = 10
• events/s = 2000 × 2 = 4,000, kafka = max(3, ceil(4000/5000)) = 3

1,000,000 residents (MODELED — the documented reference case)

• active = ceil(1000000 × 0.02) = 20,000
• PG shards = ceil(1000000/250000) = 4
• L0 shards = ceil(1000000/50000) = 20
• workers = ceil(20000/25) = 800, nodes = ceil(800/8) = 100
• events/s = 20000 × 2 = 40,000, kafka = max(3, ceil(40000/5000)) = 8

These exact numbers (4 PG shards, 20 L0 shards, 800 workers, 100 nodes, 8 Kafka partitions) are asserted in tests/scale/test_capacity_model.py so the doc and the code cannot silently drift.

Linear extrapolation to 100,000,000 residents (MODELED — NOT tested)

Pure linear scaling of the same constants (the model is linear above the single-shard floor):

• active = ceil(100000000 × 0.02) = 2,000,000
• PG shards = ceil(100000000/250000) = 400
• L0 shards = ceil(100000000/50000) = 2,000
• workers = ceil(2000000/25) = 80,000, nodes = ceil(80000/8) = 10,000
• events/s = 2000000 × 2 = 4,000,000, kafka = max(3, ceil(4000000/5000)) = 800

Summary table

Residents	Active (2%)	PG shards	L0 shards	Workers	Nodes	Kafka parts	Events/s
1,000	20	1	1	1	1	3	40
100,000	2,000	1	2	80	10	3	4,000
1,000,000	20,000	4	20	800	100	8	40,000
100,000,000	2,000,000	400	2,000	80,000	10,000	800	4,000,000

What is measured vs extrapolated

• Measured: AddNode p50 = 0.187 ms, ~5,000 ops/sec/connection (epistemic-graph/docs/benchmarks.md). This anchors OPS_PER_SEC_PER_KAFKA_PARTITION.
• Extrapolated (MODELED): every shard / worker / node / partition figure for ≥10k residents. Linear scaling is an assumption; real systems hit super-linear coordination costs (cross-shard queries, rebalancing, tail latency) that this first-order model intentionally ignores. Treat 1M as an engineering target to validate, and 100M as an order-of-magnitude sketch only.

Queue-driven dispatch stage (IMPLEMENTED — CONCEPT:AU-ORCH.dispatch.queue-agent-dispatch)

The "Workers" column above used to be aspirational on the active-concurrency axis: agent turns executed only inside the in-process asyncio scheduler (core/cognitive_scheduler.py, max_concurrent per process) on the host that accepted them. That stage is now implemented:

• agent turns ride the session-keyed agent_turns queue (AGENT_DISPATCH_BACKEND=queue; transport follows TASK_QUEUE_BACKEND);
• any host running agent-dispatch-worker claims and runs them against the shared state store (AU-OS.state.unified-durable-state-externalization), so "Workers = ceil(active / 25)" maps to a stateless dispatch-worker fleet spread across "Nodes", not to one process's coroutine cap;
• AGENT_TURNS_PARTITIONS bounds fleet-wide session concurrency on Kafka the same way the Kafka-parts column bounds event drain.

Deployment shape per row of the summary table: stateless gateways + N dispatch workers + M kg-ingest-worker processes + the listed engine/PG shards and Kafka partitions. Design, ordering and idempotency guarantees: architecture/agent_dispatch.md.

W3 deployment evidence boundary

The committed Kubernetes assets under deploy/k8s/production-cell/ include HorizontalPodAutoscaler objects for the gateway, dispatch, ingest, and analytics worker Deployments. They are IMPLEMENTED reference/staged assets: the templates intentionally are not directly applicable, and their presence does not prove that a rendered release passed compatibility checks, that an HPA is installed, or that any target is healthy. Follow the exact render/check/apply sequence in deploy/README.md. Only a bounded run against that exact release can raise the evidence level to LAB-PROVEN or LIVE; the HPA defaults do not certify the one-million workload.

The workload contract + load generator + soak/chaos harness (SCALE-P2-1)

The acceptance criterion for "1,000,000 residents" is now: sustained SLOs with bounded resource use, and no lost/duplicate/cross-tenant/falsely-completed side effects — not an infrastructure-count formula. Three artifacts implement that:

1. The contract (workload_contract.yml + workload_contract.py) — a machine-readable, validated (WorkloadContractError on any malformed/inconsistent field) definition of the workload: population, concurrency, five independent rate axes (turns/tool-calls/graph-mutations/messages/tokens per second), tenant count + Zipf skew + one deliberately oversized elephant tenant, per-agent working-set/history/media footprint, interactive/background mix, availability target + RPO/RTO, and the four SLO axes (queue/query/write/ end-to-end latency) each with p50/p95/p99/p99.9 targets. ScaledWorkload scales the population/rate axes by a scale factor for a small CI run or the full scale=1.0 — the SLO targets and per-unit sizes never scale.
2. The load generator (scripts/scale/loadgen.py) — actually GENERATES that workload: submits WorkItems (turns) through the real engine-native CAS/lease/fencing state machine (knowledge_graph/core/work_durability.py), publishes AgentBus messages with tenant namespacing, drives independent mutation/tool-call producers, and measures the four SLO axes' percentiles for real. Two engine modes: --engine mock (an in-memory FakeScaleEngine, driven by a deterministic single-threaded discrete-event simulation — immune to host CPU jitter and near-instant in real wall time, since nothing genuinely sleeps) and --engine live (the process-active epistemic-graph engine, genuine concurrent asyncio tasks against real wall-clock time — the real-hardware soak path).
3. The soak/chaos harness (tests/scale/soak/) — asserts the acceptance criterion's invariants against the CI-runnable subset (see the table below). Production-only scenarios are one executable, fail-closed exact-release campaign in test_production_certification.py; there is no mock or skip path.

What is CI-measured vs hardware-pending

Scenario	CI-runnable today	Notes
Steady-state SLO + invariant check (scaled)	Yes — test_steady_burst.py::test_steady_phase_slos_and_invariants_hold	Scaled-down population/rates; SLO targets are the full-scale ones (per-operation, not population-dependent)
Burst-on-steady + bounded backlog	Yes — test_steady_burst.py::test_burst_phase_bounded_backlog_and_slos_hold	Asserts the worker pool drains the burst, not an ever-growing queue
Worker/host loss mid-lease + crash-recovery reclaim	Yes — test_chaos_worker_and_delivery.py	Exact CAS/lease state built directly against work_durability.py, not rate-based
Duplicate/redelivered claim + idempotent re-ack	Yes — test_chaos_worker_and_delivery.py	At-least-once redelivery modeled at the claim/commit level
Timeout → retry (backoff) → DLQ	Yes — test_chaos_lifecycle_and_dlq.py	Exercises max_attempts/backoff/dead-letter exactly
Cancel mid-flight (never falsely-completes)	Yes — test_chaos_lifecycle_and_dlq.py
Hot-tenant/noisy-neighbor quota isolation	Yes — test_chaos_tenant_and_restart.py	Elephant tenant hits max_tenant_in_flight; other tenants unaffected
Full restart / cold activation (state-machine semantics)	Yes — test_chaos_tenant_and_restart.py	Engine snapshot/restore models the durable-store guarantee; NOT a claim about real 1M-scale cold-hydrate latency (see below)
Rolling upgrade (worker-pool replacement)	Yes — test_chaos_tenant_and_restart.py	Old-generation leases expire/reclaim; fencing rejects stale acks
24-72h steady + burst soak at the REAL 1M population	Production certification — test_production_certification.py	Requires a signed exact release and provisioned production-equivalent fleet; fails immediately if either is absent
Broker rebalance / live Kafka partition expansion	Production certification	Executed by the signed campaign's exact fault-hook contract
Shard split/move under concurrent writes	Production certification	Executed against the provisioned multi-shard deployment
Worker/gateway/broker/leader/node/zone loss (REAL infra)	Production certification	Uses configured real fault hooks and recovery probes
Rolling upgrade + schema migration across REAL hosts	Production certification	Bound to the exact release and migration plan
Cold activation of the ACTUAL 1,000,000 residents	Production certification	The CI test proves state-machine semantics only; the campaign measures real-scale hydration

Guardrail this table exists to enforce: a modeled capacity is never reported as a demonstrated result. Production rows are demonstrated only by signed operational evidence from the exact-release campaign.

Production guard

Reaching any of these scales requires non-toy backends. core/config.py exposes AgentConfig.assert_production_safe() (see core/profile_guard.py): under APP_PROFILE=prod it rejects graph_persistence_type of file/sqlite and a2a_broker/a2a_storage of in-memory, so a single-host config can never be mistaken for a production deployment.