Fallacies of Distributed Computing Explained
On this page
A method call that sometimes never returns
Inside one process, calling a method either returns or throws, quickly. Call the same logic on another machine and a third outcome appears: you wait, and nothing comes back. Did the other side do the work? Did the request get lost on the way there, or the response on the way back? You can't tell. That uncertainty is the heart of distributed systems.
A distributed system is a set of independent computers that work together and look like one system to their users. Every web app with a database on another server is one. They have no shared memory and no shared clock; they coordinate only by sending messages over a network, and messages can be delayed, lost, duplicated or reordered.
The eight fallacies of distributed computing
In the 1990s, engineers at Sun Microsystems, starting with L. Peter Deutsch and later James Gosling, listed assumptions newcomers make about networks. Each is false, and each causes a familiar class of bug:
| Fallacy | What really happens | Typical bug |
|---|---|---|
| The network is reliable | Packets and connections drop | No retries; one blip fails a user request |
| Latency is zero | Every call costs milliseconds or more | A loop making 200 remote calls takes seconds |
| Bandwidth is infinite | Links saturate | Returning 50 MB responses; chatty replication |
| The network is secure | Traffic can be read or forged | Plain HTTP between services, trusted internal calls |
| Topology doesn't change | Servers, IPs and routes change constantly | Hardcoded addresses break after autoscaling (Day 54) |
| There is one administrator | Many teams, providers and policies | A firewall change elsewhere breaks your calls |
| Transport cost is zero | Data transfer and serialization cost money and CPU | Surprising cloud egress bills |
| The network is homogeneous | Different hardware, OSes, protocols and versions | Works between two of your services, fails with a partner's |
Partial failure: why distributed systems are hard
On one machine, failure is usually total: the process crashes and everything stops. In a distributed system, failure is partial: one service is slow, one replica is behind, one zone is unreachable, and everything else keeps running. Worse, a slow node and a dead node look the same from outside until a timeout fires, and a timeout doesn't tell you whether the work was done.
That's why the rest of this module exists. Each topic answers a question partial failure raises:
- How do nodes agree on anything? Consensus (Day 62).
- Who's in charge? Leader election (Day 63).
- How do we let only one do something? Distributed locks (Day 64).
- How do we retry safely? Idempotency (Day 65).
- How do we change several databases together? 2PC and sagas (Day 66).
- How do we order concurrent writes without a shared clock? Vector clocks (Day 67).
- How do we spread and replicate data? Consistent hashing and quorums (Days 68 and 69).
There is no shared clock: clock skew and NTP
Every machine has its own clock, kept roughly in step by NTP. "Roughly" is the problem. Clocks drift, NTP corrections make them jump, sometimes backwards, and a VM paused for migration wakes up behind. So:
- Don't order events from different machines by their timestamps. A write stamped 10:00:00.120 on server A may have happened after one stamped 10:00:00.150 on server B. Day 67 shows what to use instead.
- Measure durations with a monotonic clock.
DateTime.UtcNowcan jump;Stopwatchin .NET only moves forward. A timeout computed from wall-clock time can fire early or never. - Allow for skew when one machine checks another's timestamp. .NET's JWT validation accepts tokens up to 5 minutes past expiry by default (
ClockSkew) for exactly this reason. Lower it if your clocks are well synchronized; don't set it to zero.
Designing with the network in mind
- Put a timeout on every remote call, and decide what happens when it fires.
- Make operations safe to retry, because you'll retry.
- Minimize round trips: batch, cache, call in parallel.
- Encrypt and authenticate internal traffic too (Day 84).
- Find services dynamically, never by fixed address.
- Measure: latency percentiles and error rates per dependency.
Check yourself
Nothing for certain. The request or the response may have been lost, or the service may still be working on it. That's why retries must be safe.
"Latency is zero." A local method call takes nanoseconds, but each remote call costs a network round trip, often a millisecond or more, plus serialization. Two hundred sequential calls add up to hundreds of milliseconds or seconds. Batch them into one call, run them in parallel, or redesign so the data arrives together.
The servers' clocks differ slightly, so timestamps from different machines don't give a reliable order. Order causally related events by sequence or version numbers, not wall-clock time.
Some components fail or slow down while others keep working. The system must keep functioning and decide what to do without knowing exactly what failed.
Each server's clock drifts, and NTP only keeps them within some milliseconds, sometimes far worse after a misconfiguration. Events a few milliseconds apart on different machines can appear in the wrong order. Use logical ordering, such as sequence numbers from one source or version vectors, when order matters.