Catalog My learning Blog
Distributed Systems Explained: CAP, Clocks and Replication · Phase 4 · Lesson 26 · 6 min read

Distributed Systems Trade-offs in Real Architectures

On this page
  1. The design exercise: a ledger and a like counter
  2. Three trade-offs that keep coming back
  3. Designing a bank ledger
  4. Designing a viral like counter
  5. What a strong design answer includes
  6. Check yourself

The design exercise: a ledger and a like counter

Before reading the walkthrough, sketch a design for each of these two workloads. For each, decide replication, consistency, how writes are acknowledged, and what happens during a partition.

  1. Ledger: a bank's account ledger. Thousands of transfers per second. Money must never be created or lost, and every transfer is audited.
  2. Like counter: likes on posts in a social app. A popular post can get hundreds of thousands of likes per minute. Users want to see the count go up.

Three trade-offs that keep coming back

The ledger and the like counter sit at opposite ends of three trade-offs: consistency versus availability, latency versus durability, and simplicity versus resilience.
The ledger and the like counter sit at opposite ends of three trade-offs: consistency versus availability, latency versus durability, and simplicity versus resilience.
  • Consistency versus availability (Days 5 and 6): during a partition, refuse or accept?
  • Latency versus durability and safety (Days 8 and 16): acknowledge before or after the write is replicated?
  • Simplicity versus resilience (Days 24 and 25): how much machinery do you add to survive failures?

Designing a bank ledger

  • Consistency: linearizable writes per account and serializable transfers. Two concurrent withdrawals must not both see the old balance. Choose CP: during a partition, the minority side refuses writes.
  • Storage: a consensus-replicated database (Spanner, CockroachDB, or PostgreSQL with synchronous replication to a standby in another zone). Acknowledge a transfer only after a majority has stored it.
  • Transfers across systems: within one database, a single transaction. Across services (ledger and card network), a saga with idempotent steps and compensations (Days 21 and 23), with every step logged.
  • Ledger design: append-only entries (debit and credit), never updates in place. Balances are derived from entries, which makes auditing and reconciliation possible.

Here's that shape in PostgreSQL. A transfer is two entries with the same transfer id that sum to zero, written in one transaction:

SQL
CREATE TABLE entries (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  transfer_id uuid        NOT NULL,
  account     text        NOT NULL,
  amount      bigint      NOT NULL,          -- cents: negative is a debit
  created_at  timestamptz NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO entries (transfer_id, account, amount) VALUES
  ('5f0c2a7e-9b1d-4c3e-8f6a-2d7b9e1c4a50', 'alice', -10000),
  ('5f0c2a7e-9b1d-4c3e-8f6a-2d7b9e1c4a50', 'bob',    10000);
COMMIT;

SELECT account, SUM(amount) AS balance FROM entries GROUP BY account;
-- reconciliation: every transfer must balance; this should return no rows
SELECT transfer_id FROM entries GROUP BY transfer_id HAVING SUM(amount) <> 0;

In testing, a deliberately broken one-sided entry showed up in the reconciliation query straight away, which is the point: the ledger can prove its own consistency. Production ledgers also keep a per-account balance row, updated in the same transaction and locked with SELECT ... FOR UPDATE, to check for overdrafts without summing every entry.

  • The cost you accept: higher write latency, and unavailability for some accounts during a severe partition. A failed transfer is better than a wrong balance.

Designing a viral like counter

  • Consistency: eventual is fine. Nobody is harmed if a count lags by a few seconds or is off by a few during a failure. Choose AP: always accept likes.
  • Writes: a "user liked post" record is idempotent per user and post (a unique key), so retries and duplicates are harmless. Counting is separate.
  • Counting hot posts: a single counter row for a viral post becomes a hot spot. Shard the counter (N sub-counters summed on read), or buffer increments in memory or a log (Kafka) and aggregate every second. A grow-only counter CRDT (Day 10) lets regions count independently and merge.
  • Reads: serve cached counts; staleness of seconds is acceptable. Show "1.2M" rather than an exact live number.
  • The cost you accept: counts that are briefly inconsistent between users and regions. In exchange: low latency, no write failures, and cheap scaling.

Here's the idempotent like record and the sharded counter together, tested in SQLite:

Python
# counter.py: a sharded like counter; each like updates one of 16 rows
import random, sqlite3
SHARDS = 16
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE likes (post_id INTEGER, user_id INTEGER, PRIMARY KEY (post_id, user_id))")
db.execute("CREATE TABLE like_counts (post_id INTEGER, shard INTEGER, n INTEGER NOT NULL DEFAULT 0, PRIMARY KEY (post_id, shard))")

def like(post_id, user_id):
    with db:
        cur = db.execute("INSERT OR IGNORE INTO likes VALUES (?, ?)", (post_id, user_id))
        if cur.rowcount == 0:
            return                                    # a retry or a double tap: already counted
        shard = random.randrange(SHARDS)              # spread the increments over 16 rows
        db.execute("INSERT INTO like_counts VALUES (?, ?, 1) "
                   "ON CONFLICT (post_id, shard) DO UPDATE SET n = n + 1", (post_id, shard))

def count(post_id):
    return db.execute("SELECT COALESCE(SUM(n), 0) FROM like_counts WHERE post_id = ?", (post_id,)).fetchone()[0]

for user in range(5000):
    like(7, user)
like(7, 42)                                           # the same user likes again
print(count(7), "likes across", db.execute("SELECT COUNT(*) FROM like_counts").fetchone()[0], "rows")
Code
5000 likes across 16 rows

The second like from user 42 changed nothing, because the insert into likes found the row already there. Each counted like updates one of 16 rows picked at random, so up to 16 writers can increment the same post without queuing on one row's lock, and a read sums 16 small rows. Real systems usually start every post with one shard and add more only to the few posts that go viral.

What a strong design answer includes

  • Requirements first: what must never happen (lost money, a double charge) and what's tolerable (a stale count).
  • An explicit choice per axis, with the reason tied to the business, not the technology.
  • Failure behavior: what users see during a partition, a node crash, a slow dependency.
  • Duplicates and retries: idempotency at every boundary.
  • Different choices for different data in the same system. A real app has both a ledger and a like counter. Choosing per data type, not per company, is the main skill this course builds.

Check yourself

A wrong balance causes real harm while a briefly stale count doesn't, so the ledger refuses writes it can't make consistent and the counter always accepts.

Shard the counter into many sub-counters on different keys and sum them on read, or buffer increments and apply them in batches, and serve cached totals to readers. A slightly stale count is fine for likes.

Entries give a complete audit trail of every movement, can be replayed to verify any balance, and avoid lost updates from concurrent overwrites, since writers only append. Balances become a derived, checkable view.

Every transfer's entries must sum to zero, so any transfer id whose entries don't sum to zero points straight at a broken or half-written transfer.

The requirements and invariants, the consistency each part needs and why, how failures and retries are handled, and the trade-offs accepted, not just a diagram of components.