Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 7 · Lesson 61 · 4 min read

Fallacies of Distributed Computing Explained

On this page
  1. A method call that sometimes never returns
  2. The eight fallacies of distributed computing
  3. Partial failure: why distributed systems are hard
  4. There is no shared clock: clock skew and NTP
  5. Designing with the network in mind
  6. Check yourself

A method call that sometimes never returns

Inside one process, calling a method either returns or throws, quickly. Call the same logic on another machine and a third outcome appears: you wait, and nothing comes back. Did the other side do the work? Did the request get lost on the way there, or the response on the way back? You can't tell. That uncertainty is the heart of distributed systems.

A distributed system is a set of independent computers that work together and look like one system to their users. Every web app with a database on another server is one. They have no shared memory and no shared clock; they coordinate only by sending messages over a network, and messages can be delayed, lost, duplicated or reordered.

The eight fallacies of distributed computing

In the 1990s, engineers at Sun Microsystems, starting with L. Peter Deutsch and later James Gosling, listed assumptions newcomers make about networks. Each is false, and each causes a familiar class of bug:

FallacyWhat really happensTypical bug
The network is reliablePackets and connections dropNo retries; one blip fails a user request
Latency is zeroEvery call costs milliseconds or moreA loop making 200 remote calls takes seconds
Bandwidth is infiniteLinks saturateReturning 50 MB responses; chatty replication
The network is secureTraffic can be read or forgedPlain HTTP between services, trusted internal calls
Topology doesn't changeServers, IPs and routes change constantlyHardcoded addresses break after autoscaling (Day 54)
There is one administratorMany teams, providers and policiesA firewall change elsewhere breaks your calls
Transport cost is zeroData transfer and serialization cost money and CPUSurprising cloud egress bills
The network is homogeneousDifferent hardware, OSes, protocols and versionsWorks between two of your services, fails with a partner's
Eight common false assumptions about networks, and the production failure each one causes.
Eight common false assumptions about networks, and the production failure each one causes.

Partial failure: why distributed systems are hard

On one machine, failure is usually total: the process crashes and everything stops. In a distributed system, failure is partial: one service is slow, one replica is behind, one zone is unreachable, and everything else keeps running. Worse, a slow node and a dead node look the same from outside until a timeout fires, and a timeout doesn't tell you whether the work was done.

That's why the rest of this module exists. Each topic answers a question partial failure raises:

  • How do nodes agree on anything? Consensus (Day 62).
  • Who's in charge? Leader election (Day 63).
  • How do we let only one do something? Distributed locks (Day 64).
  • How do we retry safely? Idempotency (Day 65).
  • How do we change several databases together? 2PC and sagas (Day 66).
  • How do we order concurrent writes without a shared clock? Vector clocks (Day 67).
  • How do we spread and replicate data? Consistent hashing and quorums (Days 68 and 69).

There is no shared clock: clock skew and NTP

Every machine has its own clock, kept roughly in step by NTP. "Roughly" is the problem. Clocks drift, NTP corrections make them jump, sometimes backwards, and a VM paused for migration wakes up behind. So:

  • Don't order events from different machines by their timestamps. A write stamped 10:00:00.120 on server A may have happened after one stamped 10:00:00.150 on server B. Day 67 shows what to use instead.
  • Measure durations with a monotonic clock. DateTime.UtcNow can jump; Stopwatch in .NET only moves forward. A timeout computed from wall-clock time can fire early or never.
  • Allow for skew when one machine checks another's timestamp. .NET's JWT validation accepts tokens up to 5 minutes past expiry by default (ClockSkew) for exactly this reason. Lower it if your clocks are well synchronized; don't set it to zero.

Designing with the network in mind

  • Put a timeout on every remote call, and decide what happens when it fires.
  • Make operations safe to retry, because you'll retry.
  • Minimize round trips: batch, cache, call in parallel.
  • Encrypt and authenticate internal traffic too (Day 84).
  • Find services dynamically, never by fixed address.
  • Measure: latency percentiles and error rates per dependency.

Check yourself

Nothing for certain. The request or the response may have been lost, or the service may still be working on it. That's why retries must be safe.

"Latency is zero." A local method call takes nanoseconds, but each remote call costs a network round trip, often a millisecond or more, plus serialization. Two hundred sequential calls add up to hundreds of milliseconds or seconds. Batch them into one call, run them in parallel, or redesign so the data arrives together.

The servers' clocks differ slightly, so timestamps from different machines don't give a reliable order. Order causally related events by sequence or version numbers, not wall-clock time.

Some components fail or slow down while others keep working. The system must keep functioning and decide what to do without knowing exactly what failed.

Each server's clock drifts, and NTP only keeps them within some milliseconds, sometimes far worse after a misconfiguration. Events a few milliseconds apart on different machines can appear in the wrong order. Use logical ordering, such as sequence numbers from one source or version vectors, when order matters.