CAP Theorem in System Design
On this page
The link between two data centers goes down
Your store runs in two data centers, and the network between them fails. Both sides are still running and still receiving customer orders. Each side can either keep accepting orders without checking with the other, or refuse orders it can't confirm. There's no third option, and that's the CAP theorem in one situation.
What is the CAP theorem?
The theorem, first stated by Eric Brewer and later proven by Seth Gilbert and Nancy Lynch, involves three properties:
- Consistency (C): every read returns the most recent write, as if there were one copy. This is not the C in ACID, which means constraints hold. Same word, different idea (Day 6 sorts them out).
- Availability (A): every request to a working node gets a non-error response.
- Partition tolerance (P): the system keeps operating when messages between nodes are lost.
The popular summary is "pick two of three". That's misleading. In any system spread over a network, partitions will happen: cables get cut, switches fail, cloud zones lose connectivity. You can't choose not to have them. So the real statement is: when a partition happens, choose consistency or availability. When there's no partition, you can have both.
A partition doesn't have to be a cut cable. A node frozen in a 20-second garbage collection pause, or a link dropping most packets, looks exactly the same to the other nodes: no answer. They can't tell slow from gone (Day 61), so they have to make the same choice.
CP vs AP systems
- CP: during a partition, the side that can't confirm it has the latest data refuses or times out. No stale or conflicting answers, at the cost of errors for some users. Systems built on consensus, such as etcd, ZooKeeper and HBase, behave this way.
- AP: both sides keep answering, possibly with stale data, and conflicting writes are reconciled after the network heals. Cassandra and Amazon's original Dynamo design are the classic examples.
"CA" only describes a single node, like one PostgreSQL server. Once data is replicated across machines, you're choosing between CP and AP whether you say so or not.
You've likely made this choice already. Synchronous replication in a database cluster leans CP: a write waits for the replica, and if the replica is unreachable, writes stall. Asynchronous replication gives up consistency instead: the primary keeps accepting writes, replicas may serve stale reads, and a failover may lose the most recent writes.
These labels describe defaults, not fixed properties. Cassandra, DynamoDB and MongoDB let you choose per request how many replicas must answer, so one database can act CP for one query and AP for the next. Day 69 shows how quorums make that dial work.
Worked example: selling the last item during a partition
Back to the two data centers and a product with one unit left.
- AP: both sides keep selling. Both might sell the last unit. When the link returns, you discover the oversell and fix it: cancel one order, apologize, offer a coupon.
- CP: only the side holding the inventory's primary copy can sell. The other side shows "temporarily unavailable" for that product until the link returns.
Many retailers choose AP for this and handle rare oversells with customer service, because a few apologetic emails cost less than refusing sales across a whole region. Amazon's Dynamo paper describes the same reasoning for the shopping cart: "add to cart" must always work, and conflicting carts are merged later (DeCandia et al., 2007). A bank ledger makes the opposite choice: an incorrect balance is worse than a brief error.
A simple rule of thumb: lean CP where correctness can't be negotiated (payments, balances, unique usernames), and AP where a little staleness is fine (likes, view counts, recommendations, product descriptions). Different data in the same system can make different choices.
Beyond CAP: PACELC and latency
CAP only talks about behavior during a partition. Most of the time there's no partition, and the trade-off you feel daily is between consistency and latency: waiting for replicas to confirm makes every write slower. The PACELC formulation names both: during a Partition choose A or C, Else choose Latency or Consistency. Day 6 looks at the consistency models in between.
Check yourself
What it does when nodes can't reach each other. Partitions happen in any networked system, so it either refuses some requests (C) or answers with possibly stale data (A).
Product reviews or recommendations can be slightly stale (AP). Charging a card or decrementing a gift card balance must not conflict (CP).
Writes stall or fail, because the primary waits for a replica acknowledgment that can't arrive. The system gives up availability for writes to avoid acknowledging data that only one node has. That's the consistency side of CAP: during a partition it prefers refusing work to risking divergent or lost data.
To the other nodes, yes. A long GC pause and a cut cable look the same from outside, so the system faces the same consistency or availability choice.
CAP only describes behavior during a partition. PACELC adds that even when there's no partition, a replicated system must trade latency against consistency, because waiting for replicas to confirm makes every write slower. That everyday trade-off often matters more than the rare partition.