Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 1 · Lesson 3 · 3 min read

Time in Distributed Systems: Clocks and Clock Skew

On this page
  1. Clock skew: two writes in the wrong order
  2. Clock drift and clock skew
  3. What NTP can and can't do
  4. Wall clock vs monotonic clock
  5. Ordering events without trusting clocks
  6. Check yourself

Clock skew: two writes in the wrong order

Server A records a user's address change at 10:00:00.120. Server B records the same user's next change at 10:00:00.095. Sorting by timestamp says B's change happened first, so the database keeps A's older address. In reality B's change came 200 ms later; B's clock was simply 225 ms behind A's. Using timestamps from different machines to decide "which happened first" quietly loses data.

Clock drift and clock skew

Computers keep time with a quartz crystal. Crystals vary with manufacturing and temperature, so clocks drift, commonly by tens of parts per million: around a second or more per day if never corrected. Two related terms:

  • Drift: how fast a clock gains or loses time.
  • Skew (offset): how far apart two clocks are at a given moment.
Each node's clock drifts at its own rate; NTP pulls them toward a reference but never makes them identical, so timestamps from different nodes can't be compared exactly.
Each node's clock drifts at its own rate; NTP pulls them toward a reference but never makes them identical, so timestamps from different nodes can't be compared exactly.

What NTP can and can't do

The Network Time Protocol periodically adjusts each machine's clock toward reference servers. Within a data center, clocks are typically kept within a millisecond or so; over the public internet, tens of milliseconds is common. But NTP:

  • can't measure network delay perfectly, so there's always residual error,
  • can step a clock backwards or forwards when the error is large,
  • can fail silently (blocked port, unreachable server), letting a clock drift for days.

Cloud providers offer more precise sources (for example AWS Time Sync and Google's internal time service), which narrow the error but never remove it.

Wall clock vs monotonic clock

ClockPurposeCan jump?
Wall clock (time of day)Timestamps shown to people, log timesYes: NTP corrections, leap seconds, manual changes
Monotonic clockMeasuring durations and timeoutsNo: only moves forward, but meaningless across machines

Use the monotonic clock for durations: time.monotonic() in Python, Stopwatch in .NET, System.nanoTime() in Java. Measuring a timeout with the wall clock breaks when NTP steps it: a timeout can fire instantly or never.

Ordering events without trusting clocks

Since wall clocks can't order events across machines, systems use other tools:

  • A single sequencer: one leader assigns increasing numbers to all writes (a database primary's log, a Kafka partition's offsets).
  • Logical clocks: counters that capture causality instead of time (Day 4).
  • Bounded uncertainty: Google's Spanner uses GPS and atomic clocks with an API (TrueTime) that returns an interval guaranteed to contain the true time, then waits out the uncertainty before committing, so commit timestamps order correctly. CockroachDB uses a similar idea with hybrid logical clocks and a configured maximum offset.

The rule of thumb: timestamps from different machines are fine for showing humans roughly when something happened, and unsafe for deciding which of two writes wins.

Check yourself

Clocks on different servers drift apart by milliseconds or more, so a write that really happened later can carry an earlier timestamp. Last-write-wins then keeps the older value and silently discards the newer one, with no error anywhere.

A monotonic clock, because it only moves forward at a steady rate. The wall clock can be stepped forward or backward by NTP or an administrator, so a duration measured with it can come out negative or wildly long.

TrueTime, backed by GPS and atomic clocks, returns an interval guaranteed to contain the real time. Spanner assigns a commit timestamp and waits until the uncertainty interval has passed before making the commit visible, so timestamps order transactions correctly.

Drift is how fast one clock gains or loses time compared to real time. Skew is the difference between two clocks at a given moment, which drift causes to grow between synchronizations.

It keeps clocks typically within a few milliseconds over a good network, but offers no hard bound: a network delay or a misconfigured server can leave a clock off by much more.