Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 1 · Lesson 1 · 3 min read

What Is a Distributed System? Partial Failure Explained

On this page
  1. A call that neither succeeded nor failed
  2. What is a distributed system?
  3. Partial failure
  4. Why build distributed systems at all
  5. How this course is organized
  6. A first design habit: three questions for every remote call
  7. Check yourself

A call that neither succeeded nor failed

Your service calls a payment service, and after five seconds the call times out. Did the payment go through? Maybe the request never arrived. Maybe it arrived, the card was charged, and the response was lost. Maybe the payment service is still processing it. On one machine, a function returns or throws. Across a network there's a third outcome: you don't know.

That uncertainty is what makes distributed systems a field of its own.

What is a distributed system?

A distributed system is a group of independent computers that communicate only by sending messages over a network and cooperate to look like one system. Three properties follow, and each removes something you take for granted on one machine:

PropertyOn one machineIn a distributed system
Shared memoryThreads see the same memoryNodes share nothing; they only exchange messages
Reliable communicationA function call arrives instantlyMessages can be delayed, lost, duplicated or reordered
A shared clockOne clock orders all eventsEach node's clock drifts; there's no global "now" (Day 3)
One machine either works or fails as a whole; a distributed system has nodes and links that fail independently, so some parts fail while others run.
One machine either works or fails as a whole; a distributed system has nodes and links that fail independently, so some parts fail while others run.

Partial failure

On a single machine, failure tends to be total: the process crashes and everything stops, which is at least clear. In a distributed system, failure is partial: one node is slow, one link drops packets, one replica is behind, and the rest carry on.

Worse, failures are often indistinguishable from each other. From the outside, these look identical:

  • the other node crashed,
  • the network between you dropped the message,
  • the other node is alive but paused (garbage collection, an overloaded host),
  • the reply was lost on the way back.

All you observe is no answer yet. A timeout is a guess, not a diagnosis.

Why build distributed systems at all

Because one machine isn't enough:

  • Scale: data or load larger than one machine can handle.
  • Availability: keep running when a machine dies.
  • Latency: serve users from nearby locations.

The price is that every interaction between parts can partially fail, and the system must keep behaving sensibly when it does.

How this course is organized

  • Days 2 to 6: the fundamental limits: unreliable communication, clocks and ordering, CAP and PACELC.
  • Days 7 to 13: replication, consistency models, quorums, conflict resolution and partitioning, ending with a Dynamo case study.
  • Days 14 to 19: agreement: the FLP result, Paxos, Raft, leader election, locks and coordination services.
  • Days 20 to 26: failures in practice: detection, idempotency, transactions, sagas, cascading failures and graceful degradation.

A first design habit: three questions for every remote call

For every remote call, ask: what if it fails, what if it's slow, and what if it succeeded but I never heard back? That third question leads straight to timeouts, retries and idempotency (Day 21), which every reliable distributed system relies on.

Check yourself

Very little: the request may not have arrived, may have been processed with the response lost, or may still be in progress.

Some components fail or slow down while others keep working, so the system must keep functioning without a clear picture of what failed.

It can only observe missing responses, and from outside a crashed node, a broken network link and a node that's just very slow all look the same. Any answer it gets back arrives too late to be certain the node is still in that state.

What happens if it's slow, what happens if it fails, and what happens if it succeeded but the response was lost. Answering them leads to timeouts, fallbacks and idempotent retries.

To get more capacity than one machine can give, to survive machine and data center failures, and to serve users from nearby locations with lower latency.