Vector Clocks and Conflict Resolution
On this page
Two carts, one customer
A shopping cart is replicated in two regions, and both accept writes (an AP design, Day 5). The customer adds a lamp on their laptop, served by region A, and a chair on their phone, served by region B, a moment apart. The regions sync later and find two different carts. Which one wins? If the answer is "the newer timestamp", the customer loses an item.
Why timestamps can't order writes
Each machine's clock drifts, and clock synchronization (NTP) keeps them close but not identical; differences of milliseconds or more are normal. Two writes milliseconds apart on different machines can carry timestamps in the wrong order. And even with perfect clocks, "later" doesn't mean "knew about the earlier one": the two cart writes were made without either seeing the other. Last-write-wins by timestamp silently discards one of them.
How vector clocks work
A vector clock tracks causality instead of time. Each version of the data carries a small map of counters, one per node:
- When a node writes, it increments its own counter in the version's clock.
- When a node receives another node's version, it keeps the higher value for each counter.
Comparing two clocks gives one of three answers:
- A happened before B if every counter in A is less than or equal to B's (and they differ). B already includes A, so keep B.
- B happened before A, the reverse.
- Concurrent: each has a counter higher than the other. Neither saw the other: a real conflict.
# vc.py: vector clocks detect whether two versions are ordered or concurrent
def happened_before(a: dict, b: dict) -> bool:
nodes = a.keys() | b.keys()
return all(a.get(n, 0) <= b.get(n, 0) for n in nodes) and a != b
def compare(a: dict, b: dict) -> str:
if happened_before(a, b):
return "a happened before b: keep b"
if happened_before(b, a):
return "b happened before a: keep a"
return "concurrent: conflict, resolve it"
base = {"A": 1} # cart created on node A
on_a = {"A": 2} # A adds a lamp (A's counter +1)
on_b = {"A": 1, "B": 1} # B, unaware of A's change, adds a chair
merged = {"A": 2, "B": 2} # after merging both, written on Bcompare(base, on_a) -> a happened before b: keep b
compare(on_a, on_b) -> concurrent: conflict, resolve it
compare(on_b, merged) -> a happened before b: keep bThe lamp and chair versions are correctly flagged as concurrent. Amazon's 2007 Dynamo paper used vector clocks for exactly this shopping-cart case.
Conflict resolution: last-write-wins, merges and CRDTs
Vector clocks detect conflicts; resolving them is a business decision:
| Strategy | How | Good for |
|---|---|---|
| Last write wins | Keep the later timestamp, discard the other | Data where losing an update is acceptable (a "last seen" time) |
| Merge | Combine both (union of cart items) | Sets, counters, lists |
| Application or user decides | Store both versions as siblings; resolve on next read | Documents, where only a person knows |
| CRDTs | Data types designed so concurrent updates always merge deterministically | Counters, sets, collaborative text |
CRDTs (conflict-free replicated data types) deserve a mention: a grow-only counter or an observed-remove set merges concurrent updates automatically, without conflicts. Collaborative editors and some databases (Redis Enterprise's active-active, Riak) use them.
Last-write-wins is more common than vector clocks in practice, and it's the default in Cassandra, where each write carries a timestamp. Know what that means: if one node's clock runs two seconds fast, its writes beat genuinely later writes from other nodes for those two seconds, and the later ones are silently dropped (Day 61). Fine for a "last seen" time; not for anything a user would notice losing.
Two related tools you'll see named: Lamport clocks, a single counter that gives every event an order consistent with causality but can't detect concurrency, and hybrid logical clocks, which combine physical time with a logical counter so timestamps stay close to real time while respecting causality. CockroachDB uses the latter.
Vector clocks vs optimistic concurrency
Optimistic concurrency (a version column checked on update) is the one-node relative: it detects that someone else changed the row since you read it and rejects your write. Vector clocks extend that idea to many nodes accepting writes independently.
Check yourself
Clock skew. Cassandra resolves conflicts by write timestamp, so a node with a fast clock wins over later writes. Fix NTP and check whether last-write-wins suits that data.
Concurrent. One clock comes after another only if all of its counters are greater than or equal to the other's. Here the first has a higher A and the second a higher B, so neither write saw the other. The system must keep both versions and resolve the conflict rather than pick one silently.
Merge them, usually by taking the union of the items, so neither addition is lost. The known side effect is that an item removed in one version can reappear, which is why designs track removals explicitly, for example with an observed-remove set CRDT.
It silently discards every concurrent write except the one with the latest timestamp, and clock skew can make the "latest" write the wrong one. It's acceptable for data where losing an update doesn't matter, like a last-seen time, but not for carts, balances or documents.
A conflict-free replicated data type: a data structure designed so that concurrent updates on different replicas always merge to the same result, whatever order they arrive in. Counters, sets and collaborative text editors use CRDTs to avoid manual conflict resolution.