What Is a Distributed System? Partial Failure Explained
On this page
A call that neither succeeded nor failed
Your service calls a payment service, and after five seconds the call times out. Did the payment go through? Maybe the request never arrived. Maybe it arrived, the card was charged, and the response was lost. Maybe the payment service is still processing it. On one machine, a function returns or throws. Across a network there's a third outcome: you don't know.
That uncertainty is what makes distributed systems a field of its own.
What is a distributed system?
A distributed system is a group of independent computers that communicate only by sending messages over a network and cooperate to look like one system. Three properties follow, and each removes something you take for granted on one machine:
| Property | On one machine | In a distributed system |
|---|---|---|
| Shared memory | Threads see the same memory | Nodes share nothing; they only exchange messages |
| Reliable communication | A function call arrives instantly | Messages can be delayed, lost, duplicated or reordered |
| A shared clock | One clock orders all events | Each node's clock drifts; there's no global "now" (Day 3) |
Partial failure
On a single machine, failure tends to be total: the process crashes and everything stops, which is at least clear. In a distributed system, failure is partial: one node is slow, one link drops packets, one replica is behind, and the rest carry on.
Worse, failures are often indistinguishable from each other. From the outside, these look identical:
- the other node crashed,
- the network between you dropped the message,
- the other node is alive but paused (garbage collection, an overloaded host),
- the reply was lost on the way back.
All you observe is no answer yet. A timeout is a guess, not a diagnosis.
Why build distributed systems at all
Because one machine isn't enough:
- Scale: data or load larger than one machine can handle.
- Availability: keep running when a machine dies.
- Latency: serve users from nearby locations.
The price is that every interaction between parts can partially fail, and the system must keep behaving sensibly when it does.
How this course is organized
- Days 2 to 6: the fundamental limits: unreliable communication, clocks and ordering, CAP and PACELC.
- Days 7 to 13: replication, consistency models, quorums, conflict resolution and partitioning, ending with a Dynamo case study.
- Days 14 to 19: agreement: the FLP result, Paxos, Raft, leader election, locks and coordination services.
- Days 20 to 26: failures in practice: detection, idempotency, transactions, sagas, cascading failures and graceful degradation.
A first design habit: three questions for every remote call
For every remote call, ask: what if it fails, what if it's slow, and what if it succeeded but I never heard back? That third question leads straight to timeouts, retries and idempotency (Day 21), which every reliable distributed system relies on.
Check yourself
Very little: the request may not have arrived, may have been processed with the response lost, or may still be in progress.
Some components fail or slow down while others keep working, so the system must keep functioning without a clear picture of what failed.
It can only observe missing responses, and from outside a crashed node, a broken network link and a node that's just very slow all look the same. Any answer it gets back arrives too late to be certain the node is still in that state.
What happens if it's slow, what happens if it fails, and what happens if it succeeded but the response was lost. Answering them leads to timeouts, fallbacks and idempotent retries.
To get more capacity than one machine can give, to survive machine and data center failures, and to serve users from nearby locations with lower latency.