Catalog My learning Blog
Scalability Patterns: Load Balancing, Caching, Sharding · Phase 4 · Lesson 22 · 13 min read

Scaling from 1,000 to 10 Million Users

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: One System, Five Stages, and the Constraint at Each
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Combine every decision this course covered, load balancing layer, distribution algorithm, caching pattern, invalidation strategy, shard key, replication topology, and hot spot mitigation, into one ordered progression, and identify at each stage which constraint actually forces the next change.

Concept Explanation

This is a synthesis lesson rather than new material. The purpose is a repeatable sequence for scaling a system, and the most important property of that sequence is its order, because almost every expensive architectural mistake is a step taken before the one that should have preceded it. Sharding before adding read replicas, caching before fixing an index, adding regions before making one region independent, and building a load balancing tier before measuring where the bottleneck actually is are all the same error: applying a later stage remedy to an earlier stage problem, thereby paying permanent complexity for a benefit that a simpler change would have provided.

The stages are driven by constraints, not by user counts, and the user counts below are only a rough calibration. At small scale a single application server and a single database is correct, and the useful work is measurement rather than architecture: know your per-request cost, your slowest queries, and your read-to-write ratio, because every later decision is derived from those numbers. The first real constraint is usually a single machine serving both application and database, and the first change is to separate them, which is vertical partitioning in its simplest form and buys a great deal for almost no complexity. Then application servers become the bottleneck, which is solved by running several behind a load balancer, and at that moment two decisions from this course arrive together: which layer the load balancer operates at, and whether sessions are externalized so that any instance can serve any request. Externalizing session state here, rather than reaching for sticky routing, is what keeps every later option open.

The next constraint is read load on the database, and the correct response is caching for the expensive repeated reads plus read replicas for general read volume, in that order because caching is cheaper and often sufficient. This is where cache-aside, TTLs with jitter, and negative caching are introduced, and where the discipline of measuring hit rate per key pattern begins. Static and cacheable content moves to a CDN in the same period, which typically removes a large fraction of total requests from the origin for very little work. After that, write load or dataset size eventually exceeds one primary, and only then is sharding justified, with the shard key chosen by the four properties from this course and the logical-shard indirection built in from the start rather than retrofitted. Replication continues alongside, because as shard count rises, per-shard availability must rise with it.

At the largest stage, the problems stop being about aggregate capacity and become about distribution and failure. Hot keys and hot partitions appear because the access distribution is a power law and always was, and they are addressed with local caching, splitting, coalescing, and per-key limits rather than with more hardware. Multi-region deployment arrives when latency or availability requirements demand it, and it is done by making regions independent failure domains, serving reads locally and routing writes to the region that owns the data. Admission control and load shedding become mandatory rather than optional, because at this size something is always partially broken and the system must degrade along boundaries someone chose. And the operational practices, per-shard monitoring, realistic load testing, and rehearsed failure, are what keep all of the above from being theoretical.

The transferable skill is the diagnostic order, not the memory of any single technique. Faced with a scaling problem, the sequence is: measure what is actually saturated, confirm the cheapest applicable remedy has been exhausted, apply the smallest change that addresses the measured constraint, and verify the constraint moved. Almost everything in this course is an answer to a specific measured constraint, and applying any of it without that measurement is how systems acquire complexity that solves nothing.

StageConstraint That Forces ItChangeCourse Lessons
One serverNone yetMeasure per-request cost and read-to-write ratioPrerequisite for every later decision
Split app and databaseOne machine serving bothSeparate the tiers; vertical partitioningLesson 12
Multiple app serversApplication CPU saturatedLoad balancer; externalize session stateLessons 1 to 4
Cache and CDNRepeated expensive reads; static asset volumeCache-aside with TTL jitter; CDN for static contentLessons 6 to 11
Read replicasRead volume beyond one databaseReplicas plus read-your-writes handlingLesson 16
ShardWrite throughput or dataset size beyond one primaryShard key by the four properties; logical shard indirectionLessons 12 to 15
Hot spot mitigationOne shard or key saturated while the fleet idlesLocal caching, splitting, coalescing, per-key limitsLessons 17 to 19
Multi-regionGlobal latency or regional availability requirementsIndependent regions; reads local, writes homeLesson 5
Admission controlOverload collapsing goodputBounded queues, deadlines, prioritized sheddingLesson 19

Worked Example: One System, Five Stages, and the Constraint at Each

The progression below follows a single marketplace product. At each stage the note that matters most is not what was built but what was measured first, and what was deliberately not built yet.

Code
# === STAGE 1: ~1,000 users ===============================================
# One app server, one Postgres, one machine. This is correct.
# Work done: instrument everything.
#   p50/p99 per endpoint, slow query log on, per-request CPU measured,
#   read:write ratio measured (turns out to be 40:1)
# NOT done: no cache, no LB, no shards. There is no constraint yet.

# === STAGE 2: ~50,000 users ==============================================
# Constraint MEASURED: app CPU at 85% during peak; db at 20%.
# Change: 4 app instances behind an L7 load balancer.
#   algorithm: round robin (requests are homogeneous - verified, not assumed)
#   health:    readiness probe checks LOCAL state only; panic threshold 50%
#   sessions:  moved to a shared store on day one
# Why sessions first: sticky routing would have made every later step
# harder - deploys disruptive, scaling events lossy, load never truly even.

# === STAGE 3: ~500,000 users =============================================
# Constraint MEASURED: db CPU at 78%, dominated by 3 query shapes.
# Step 3a (cheapest first): add the missing index on listings(seller_id,
#          created_at). Two of the three query shapes disappear.
# Step 3b: cache-aside on the remaining expensive aggregate.
#          ttl = 300 * (0.9 + 0.2*rand())      # jitter, from day one
#          negative caching with a 30s sentinel  # penetration closed
#          hit rate tracked PER KEY PREFIX, not globally
# Step 3c: CDN for images and static assets. Removes 71% of all requests
#          from the origin for a day of work.
# Step 3d: 2 read replicas + a read-from-primary window after writes.
# NOT done: no sharding. Writes are at 400/sec against a primary that
# handles 3,000/sec. Sharding here would be pure cost.

# === STAGE 4: ~3,000,000 users ===========================================
# Constraint MEASURED: writes at 2,900/sec vs a 3,000/sec primary ceiling,
# and the listings table is 3.1 TB. BOTH sharding triggers, finally real.
# Shard key evaluation (the four properties):
#   listing_id  - alignment FAILS: 64% of reads filter by seller
#   category_id - cardinality FAILS: ~200 values, permanent hot shards
#   seller_id   - cardinality GOOD, alignment GOOD, immutable GOOD,
#                 distribution UNEVEN (top sellers are 1000x median)
#   -> seller_id, compound with created_at for ordering.
#      Distribution skew is a hot-shard problem WITH known mitigations;
#      alignment and immutability failures have none.
# Built in from the start: 4096 logical shards -> 8 physical machines,
# via a directory. Each shard has 2 replicas. 24 instances total.
# Migration: dual-write, backfill, shadow-read verify, cut over by
# config in slices, soak a full week, THEN stop dual-writing.

# === STAGE 5: ~10,000,000 users ==========================================
# Constraint MEASURED: p99 at 890ms while p50 holds at 4ms.
#   max_shard_load / median_shard_load = 8.2
#   per-key sampling: 3 seller accounts are 61% of one shard's traffic
# This is a hot-spot problem, NOT a capacity problem. More machines would
# have changed nothing.
# Mitigations, in order of leverage:
#   1. local in-process cache, 2s TTL, on hot seller storefronts
#      -> 94,000 reads/sec becomes ~100/sec at the shared store
#   2. request coalescing on every cache miss path
#   3. per-key rate limits so one seller cannot consume a shard
#   4. the 3 largest sellers moved to dedicated logical shards via the
#      directory - which cost nothing, because the indirection layer was
#      built in at Stage 4 instead of retrofitted
#   5. admission control: bounded queues, request deadlines, shed batch
#      and retries before user-facing traffic
# Then: a second region, as INDEPENDENT failure domains. Reads local,
# writes routed to the seller's home region. No synchronous cross-region
# call on any request path.

# === The diagnostic order, which is the actual takeaway =================
#   1. Measure what is saturated. Not what feels slow.
#   2. Exhaust the cheaper remedy first (index before cache, cache before
#      replica, replica before shard).
#   3. Apply the smallest change that addresses the MEASURED constraint.
#   4. Verify the constraint moved. If it did not, you diagnosed wrong.
# Every expensive mistake in scaling is a stage applied out of order.
A five-stage scaling progression for one system, each stage labelled with the measured constraint that forced it and the specific change made, showing the architecture growing from a single server to a sharded, replicated, multi-region fleet with hot spot mitigations layered on last.
A five-stage scaling progression for one system, each stage labelled with the measured constraint that forced it and the specific change made, showing the architecture growing from a single server to a sharded, replicated, multi-region fleet with hot spot mitigations layered on last.

Real-World Examples

  • Instagram reached tens of millions of users on a famously small engineering team by following close to this order: it scaled PostgreSQL vertically and with replicas first, moved static content to a CDN, cached aggressively with memcached, and only then introduced logical sharding with the shard identifier embedded in object ids. The published accounts consistently emphasize choosing boring, well-understood technology and deferring each step until a measurement demanded it.
  • Shopify has published extensively about its pods architecture, in which the platform is divided into isolated units each containing its own database and supporting services with shops assigned to pods, chosen specifically so that a failure or a hot shop is contained within one pod rather than affecting the whole platform, and so that capacity can be added by adding pods. It is a production example of shard-level isolation being adopted for blast radius reasons alongside capacity ones, which is the argument this course makes for treating shards as independent failure domains.
Key TakeawaysEvery expensive scaling mistake is a stage applied out of order: sharding before adding replicas, caching before fixing an index, adding regions before making one region independent. The diagnostic sequence is always the same, measure what is actually saturated, exhaust the cheaper remedy, apply the smallest change that addresses the measured constraint, and verify the constraint moved. Complexity added without a measurement behind it is complexity that solves nothing and has to be operated forever.

Review Questions & Answers

This is a self-directed synthesis exercise, so there is no single correct answer, and the value is in the discipline of naming the measured constraint before naming a remedy. The common outcome is discovering that a system has adopted a later-stage technique, most often sharding or an elaborate caching layer, without having exhausted an earlier one, or that nobody can state the current bottleneck in measured terms at all, which means any architectural change would be a guess. The useful output is a short list: what resource is closest to saturation, what the cheapest remaining remedy for it is, and what specific measurement would indicate that the next stage has genuinely become necessary.

Because each earlier step is strictly cheaper in complexity and often removes the need for the later one entirely. A CDN and a cache require no change to the data model, no key decisions, and no query restrictions, and they frequently remove the large majority of read load, which can defer replicas indefinitely. Read replicas require no data model change either and solve read throughput completely, at the cost of replication lag and read-your-writes handling, which is a bounded and well-understood problem. Sharding is the only step that permanently constrains the data model, makes cross-shard queries expensive, complicates transactions, and multiplies every future schema change, and it solves a problem, write throughput and dataset size, that the earlier steps genuinely cannot address. Taking it early means paying its permanent costs to solve a problem a cache would have solved, which is the most common and most expensive ordering mistake in the sequence.

A moving tail over a stationary median means a subset of requests is slow, not the system, and the per-shard and per-key views confirm it is one shard and a handful of keys. Doubling the shard count redistributes the key space, but each hot key still hashes to exactly one shard, so the hot keys arrive in their new homes exactly as hot as before. The fleet cost doubles, the cross-shard query cost rises because fan-out now touches twice as many shards, and the actual problem is untouched. The correct sequence is to identify the specific hot keys by sampling, apply local in-process caching for the read-hot ones, which typically reduces shared store load by several orders of magnitude, add request coalescing on the miss path, apply per-key rate limits so one key cannot consume a shard capacity and harm its neighbours, and where the skew is tenant-shaped, move the largest tenants to dedicated logical shards through the directory layer. Only after the distribution has been addressed does adding aggregate capacity make sense, and by then it usually is not needed.

The logical shard indirection: hashing keys into a large fixed number of logical shards and mapping those to physical machines through a small directory, rather than hashing keys directly to machines. At the moment sharding is introduced, it is a handful of lines and one configuration table. Without it, every subsequent capacity change remaps a large fraction of all keys and requires a full global data reshuffle with no safe intermediate state, so growing the fleet becomes a project rather than an operation. With it, adding capacity means copying whole logical shards, verifying each one, and flipping its mapping entry, one shard at a time and reversibly. It is also what makes moving a single large tenant to dedicated capacity a configuration change at stage five rather than a redesign, which is exactly the hot spot mitigation the system will need. The general lesson is that the cheap thing to build early is the indirection that preserves future options, not the capacity itself.

Knowing the techniques is knowing what each one does once you have decided to apply it. Being able to scale a system is knowing which constraint is actually binding right now, which is a measurement question rather than a knowledge question, and then choosing the smallest change that relieves it. Nearly every technique in this course is correct in some context and actively harmful in another: caching helps enormously with a high hit rate and makes things slower with a low one, sharding is essential past a write ceiling and pure cost before it, affinity routing is the point of consistent hashing and the cause of hot keys, and load-aware balancing is an improvement over round robin and a mechanism for routing traffic into a failing server. The skill is holding the diagnostic order, measure, exhaust the cheaper remedy, change the smallest thing, verify the constraint moved, rather than reaching for whichever technique is most familiar or most recently read about.