Scaling and Operating a CQRS System
On this page
Learning Objective
Scale the read and write sides independently according to their different characteristics, build the operational tooling that a CQRS system requires rather than assumes, and run the routine procedures, rebuilds, deploys, and incident response, without serving wrong answers.
Concept Explanation
The strongest operational argument for CQRS is that the two sides have different scaling profiles and can finally be scaled independently. Read volume in most systems exceeds write volume by one to three orders of magnitude, and reads are easy to scale horizontally because read models are derived, disposable, and require no coordination: add replicas, add caches, add entire additional read stores shaped for new access patterns, none of which affects correctness. The write side scales differently and less easily, because it must enforce invariants, which means partitioning by aggregate and accepting that a single hot aggregate is a serialization point no amount of hardware removes.
That asymmetry shapes the capacity plan. Scaling reads means more replicas and more projections, and the limiting factor is usually projection throughput rather than query throughput, because every read model must keep up with the write stream regardless of how many replicas serve it. Scaling writes means sharding by aggregate id so that unrelated aggregates never contend, and the honest constraint is that commands against one aggregate serialize by design. When one aggregate becomes a bottleneck, the fix is domain modeling, splitting the aggregate or restructuring so that contention disappears, rather than infrastructure.
Projection throughput deserves its own attention because it is the component most likely to be the constraint. Projections can usually be parallelized by partitioning on the aggregate id, which preserves per-aggregate ordering while allowing many workers, and they can be sped up by batching writes rather than committing per event. The limit is any projection that requires global ordering across all aggregates, such as a running sequence number or a global rank, which cannot be parallelized and should be recognized as a design decision with a throughput ceiling attached.
The tooling a CQRS system needs is substantial and largely must be built, which is the cost teams most often fail to budget for. At minimum: per-projection lag dashboards in events and in seconds; a rebuild command that can be run safely on any projection and is exercised regularly rather than written once and never tested; a dead letter queue with an inspection and replay interface for individual failed events; a reconciliation job that samples read-model rows and compares them against the authoritative source to catch silent drift; and a runbook that says, for each projection, what its expected lag is, who owns it, and what the product does when it is exceeded.
Two routine procedures need explicit design. Deploying a projection change usually means a rebuild, which for a large history should be a shadow build with an atomic swap rather than an in-place truncate, and it means the deployment process must tolerate old and new projection code running simultaneously during the rollout. Incident response needs a triage order specific to this architecture: check whether the write side accepted the write, then whether the event was emitted, then whether the relay published it, then whether the projection consumed it, then whether the handler applied it. That chain is the single most useful debugging artifact in a CQRS system, and a system that cannot be walked along it quickly will have long incidents.
| Concern | Write side | Read side |
|---|---|---|
| Typical volume | Lower, often by orders of magnitude | Higher, and growing with every new view |
| How it scales | Partition by aggregate; hot aggregates serialize | Replicas, caches, and additional stores, freely |
| Limiting factor | Contention on a single aggregate | Projection throughput keeping up with the write stream |
| Failure impact | Writes rejected; the system stops accepting change | Stale or unavailable views; the truth is unaffected |
| Recovery | Restore from backup; data loss is real | Rebuild from the authoritative source; no data loss |
| Deploy risk | Schema migration on authoritative data | Rebuild with a shadow table and an atomic swap |
Worked Example: A Parallel Projection Runner With Safe Ordering
Parallelism is partitioned by aggregate id so that events for one aggregate always land on the same worker in order, which is the only ordering guarantee most projections actually require.
// Ordering is needed per aggregate, not globally. Hashing the aggregate id to
// a worker preserves exactly that, and lets N workers run concurrently.
function workerFor(aggregateId, workerCount) {
return hash32(aggregateId) % workerCount;
}
async function runPartitionedProjection(name, handlers, { workers = 8, batch = 500 }) {
await Promise.all(
Array.from({ length: workers }, (_, id) => runWorker(name, handlers, id, workers, batch)),
);
}
async function runWorker(name, handlers, workerId, workerCount, batchSize) {
const checkpointKey = `${name}:w${workerId}`; // one checkpoint per worker
let { position } = await db.checkpoints.get(checkpointKey) ?? { position: 0 };
for (;;) {
const events = (await store.readAll({ after: position, limit: batchSize }))
.filter((e) => workerFor(e.aggregateId, workerCount) === workerId);
if (events.length === 0) { await sleep(200); continue; }
const tx = await db.begin();
try {
for (const event of events) {
const handler = handlers[event.type];
if (handler) await handler(tx, event);
position = event.globalPosition;
}
await tx.query(
`INSERT INTO projection_checkpoints (projection_name, position) VALUES ($1, $2)
ON CONFLICT (projection_name) DO UPDATE SET position = $2, updated_at = now()`,
[checkpointKey, position],
);
await tx.commit(); // batch commit, not per event
} catch (err) {
await tx.rollback();
await deadLetter(name, events, err); // one bad batch must not stop the worker
position = events[events.length - 1].globalPosition;
}
}
}
// Reported lag is the MINIMUM position across workers, because the projection
// is only caught up to the point every partition has passed.
async function projectionLag(name, workerCount) {
const positions = await Promise.all(
Array.from({ length: workerCount }, (_, i) => db.checkpoints.get(`${name}:w${i}`)));
const slowest = Math.min(...positions.map((p) => p?.position ?? 0));
return (await store.headPosition()) - slowest;
}Real-World Examples
- Kafka Streams and Apache Flink both implement the partition-by-key parallelism model this lesson describes, guaranteeing per-key ordering while scaling out across many tasks, and both expose consumer lag as the primary operational metric. Their state store and changelog design is essentially a productized projection with checkpointing, which is a fair indication that hand-rolled projection runners converge on the same structure once they meet production requirements.
- The industry practice of running periodic reconciliation between a system of record and its derived indexes is widespread in commerce and payments, where a nightly job compares totals or samples rows between the authoritative database and the search or reporting store. It exists because silent projection drift, an event type handled incorrectly or a message lost before outbox patterns were adopted, produces views that look healthy and are wrong, and no amount of lag monitoring detects that class of error.
Review Questions & Answers
Because a read of the projection can touch data maintained by any worker, so the view is only as fresh as the least advanced partition. If seven workers are at position 10,000 and one is at 4,000, then any aggregate belonging to the lagging worker has a view that is 6,000 events stale, and a caller asking whether its write is visible has no way to know which worker owns its aggregate. Reporting the maximum or the average would make the projection appear healthy while a portion of the data is badly behind, which is precisely the silent staleness that operational monitoring exists to prevent.
Because commands against one aggregate must serialize in order to enforce its invariants atomically, and that serialization is the mechanism of correctness rather than an implementation limitation. No amount of additional hardware changes it, and removing it would remove the guarantee. The real fix is to change the model so the contention does not exist: split the aggregate if its parts have independent invariants, restructure so that high-frequency changes append to separate aggregates and are aggregated eventually, or replace a shared counter with per-partition counters that are summed on read. Each of those is a domain design decision.
Did the write side accept the change, was an event emitted for it, did the relay publish that event, did the projection consume it, and did the handler apply it correctly. It matters because in a system with this shape the symptom is almost always the same, a user reports that something they did is not visible, while the cause can be at any of five points with completely different owners and remedies. Without the chain, responders check whichever component they know best and incidents run long. With it, each step is a single query or metric that eliminates a stage, and the cause is usually isolated in minutes.
Because lag monitoring measures whether events are being processed, not whether they are being processed correctly. A projection with a handler that computes a total wrongly, ignores a newly added event type, or applies a status transition incorrectly will show zero lag and perfect health while producing a view that is quietly wrong. Reconciliation compares the derived answer against the authoritative source, which is the only check that tests the derivation itself. It should sample rather than compare everything, run continuously at a low rate, and alert on any mismatch at all, since a single genuine mismatch usually indicates a systematic handler bug rather than a one-off.