Scaling from 1,000 to 10 Million Users
On this page
Learning Objective
Combine every decision this course covered, load balancing layer, distribution algorithm, caching pattern, invalidation strategy, shard key, replication topology, and hot spot mitigation, into one ordered progression, and identify at each stage which constraint actually forces the next change.
Concept Explanation
This is a synthesis lesson rather than new material. The purpose is a repeatable sequence for scaling a system, and the most important property of that sequence is its order, because almost every expensive architectural mistake is a step taken before the one that should have preceded it. Sharding before adding read replicas, caching before fixing an index, adding regions before making one region independent, and building a load balancing tier before measuring where the bottleneck actually is are all the same error: applying a later stage remedy to an earlier stage problem, thereby paying permanent complexity for a benefit that a simpler change would have provided.
The stages are driven by constraints, not by user counts, and the user counts below are only a rough calibration. At small scale a single application server and a single database is correct, and the useful work is measurement rather than architecture: know your per-request cost, your slowest queries, and your read-to-write ratio, because every later decision is derived from those numbers. The first real constraint is usually a single machine serving both application and database, and the first change is to separate them, which is vertical partitioning in its simplest form and buys a great deal for almost no complexity. Then application servers become the bottleneck, which is solved by running several behind a load balancer, and at that moment two decisions from this course arrive together: which layer the load balancer operates at, and whether sessions are externalized so that any instance can serve any request. Externalizing session state here, rather than reaching for sticky routing, is what keeps every later option open.
The next constraint is read load on the database, and the correct response is caching for the expensive repeated reads plus read replicas for general read volume, in that order because caching is cheaper and often sufficient. This is where cache-aside, TTLs with jitter, and negative caching are introduced, and where the discipline of measuring hit rate per key pattern begins. Static and cacheable content moves to a CDN in the same period, which typically removes a large fraction of total requests from the origin for very little work. After that, write load or dataset size eventually exceeds one primary, and only then is sharding justified, with the shard key chosen by the four properties from this course and the logical-shard indirection built in from the start rather than retrofitted. Replication continues alongside, because as shard count rises, per-shard availability must rise with it.
At the largest stage, the problems stop being about aggregate capacity and become about distribution and failure. Hot keys and hot partitions appear because the access distribution is a power law and always was, and they are addressed with local caching, splitting, coalescing, and per-key limits rather than with more hardware. Multi-region deployment arrives when latency or availability requirements demand it, and it is done by making regions independent failure domains, serving reads locally and routing writes to the region that owns the data. Admission control and load shedding become mandatory rather than optional, because at this size something is always partially broken and the system must degrade along boundaries someone chose. And the operational practices, per-shard monitoring, realistic load testing, and rehearsed failure, are what keep all of the above from being theoretical.
The transferable skill is the diagnostic order, not the memory of any single technique. Faced with a scaling problem, the sequence is: measure what is actually saturated, confirm the cheapest applicable remedy has been exhausted, apply the smallest change that addresses the measured constraint, and verify the constraint moved. Almost everything in this course is an answer to a specific measured constraint, and applying any of it without that measurement is how systems acquire complexity that solves nothing.
| Stage | Constraint That Forces It | Change | Course Lessons |
|---|---|---|---|
| One server | None yet | Measure per-request cost and read-to-write ratio | Prerequisite for every later decision |
| Split app and database | One machine serving both | Separate the tiers; vertical partitioning | Lesson 12 |
| Multiple app servers | Application CPU saturated | Load balancer; externalize session state | Lessons 1 to 4 |
| Cache and CDN | Repeated expensive reads; static asset volume | Cache-aside with TTL jitter; CDN for static content | Lessons 6 to 11 |
| Read replicas | Read volume beyond one database | Replicas plus read-your-writes handling | Lesson 16 |
| Shard | Write throughput or dataset size beyond one primary | Shard key by the four properties; logical shard indirection | Lessons 12 to 15 |
| Hot spot mitigation | One shard or key saturated while the fleet idles | Local caching, splitting, coalescing, per-key limits | Lessons 17 to 19 |
| Multi-region | Global latency or regional availability requirements | Independent regions; reads local, writes home | Lesson 5 |
| Admission control | Overload collapsing goodput | Bounded queues, deadlines, prioritized shedding | Lesson 19 |
Worked Example: One System, Five Stages, and the Constraint at Each
The progression below follows a single marketplace product. At each stage the note that matters most is not what was built but what was measured first, and what was deliberately not built yet.
# === STAGE 1: ~1,000 users ===============================================
# One app server, one Postgres, one machine. This is correct.
# Work done: instrument everything.
# p50/p99 per endpoint, slow query log on, per-request CPU measured,
# read:write ratio measured (turns out to be 40:1)
# NOT done: no cache, no LB, no shards. There is no constraint yet.
# === STAGE 2: ~50,000 users ==============================================
# Constraint MEASURED: app CPU at 85% during peak; db at 20%.
# Change: 4 app instances behind an L7 load balancer.
# algorithm: round robin (requests are homogeneous - verified, not assumed)
# health: readiness probe checks LOCAL state only; panic threshold 50%
# sessions: moved to a shared store on day one
# Why sessions first: sticky routing would have made every later step
# harder - deploys disruptive, scaling events lossy, load never truly even.
# === STAGE 3: ~500,000 users =============================================
# Constraint MEASURED: db CPU at 78%, dominated by 3 query shapes.
# Step 3a (cheapest first): add the missing index on listings(seller_id,
# created_at). Two of the three query shapes disappear.
# Step 3b: cache-aside on the remaining expensive aggregate.
# ttl = 300 * (0.9 + 0.2*rand()) # jitter, from day one
# negative caching with a 30s sentinel # penetration closed
# hit rate tracked PER KEY PREFIX, not globally
# Step 3c: CDN for images and static assets. Removes 71% of all requests
# from the origin for a day of work.
# Step 3d: 2 read replicas + a read-from-primary window after writes.
# NOT done: no sharding. Writes are at 400/sec against a primary that
# handles 3,000/sec. Sharding here would be pure cost.
# === STAGE 4: ~3,000,000 users ===========================================
# Constraint MEASURED: writes at 2,900/sec vs a 3,000/sec primary ceiling,
# and the listings table is 3.1 TB. BOTH sharding triggers, finally real.
# Shard key evaluation (the four properties):
# listing_id - alignment FAILS: 64% of reads filter by seller
# category_id - cardinality FAILS: ~200 values, permanent hot shards
# seller_id - cardinality GOOD, alignment GOOD, immutable GOOD,
# distribution UNEVEN (top sellers are 1000x median)
# -> seller_id, compound with created_at for ordering.
# Distribution skew is a hot-shard problem WITH known mitigations;
# alignment and immutability failures have none.
# Built in from the start: 4096 logical shards -> 8 physical machines,
# via a directory. Each shard has 2 replicas. 24 instances total.
# Migration: dual-write, backfill, shadow-read verify, cut over by
# config in slices, soak a full week, THEN stop dual-writing.
# === STAGE 5: ~10,000,000 users ==========================================
# Constraint MEASURED: p99 at 890ms while p50 holds at 4ms.
# max_shard_load / median_shard_load = 8.2
# per-key sampling: 3 seller accounts are 61% of one shard's traffic
# This is a hot-spot problem, NOT a capacity problem. More machines would
# have changed nothing.
# Mitigations, in order of leverage:
# 1. local in-process cache, 2s TTL, on hot seller storefronts
# -> 94,000 reads/sec becomes ~100/sec at the shared store
# 2. request coalescing on every cache miss path
# 3. per-key rate limits so one seller cannot consume a shard
# 4. the 3 largest sellers moved to dedicated logical shards via the
# directory - which cost nothing, because the indirection layer was
# built in at Stage 4 instead of retrofitted
# 5. admission control: bounded queues, request deadlines, shed batch
# and retries before user-facing traffic
# Then: a second region, as INDEPENDENT failure domains. Reads local,
# writes routed to the seller's home region. No synchronous cross-region
# call on any request path.
# === The diagnostic order, which is the actual takeaway =================
# 1. Measure what is saturated. Not what feels slow.
# 2. Exhaust the cheaper remedy first (index before cache, cache before
# replica, replica before shard).
# 3. Apply the smallest change that addresses the MEASURED constraint.
# 4. Verify the constraint moved. If it did not, you diagnosed wrong.
# Every expensive mistake in scaling is a stage applied out of order.Real-World Examples
- Instagram reached tens of millions of users on a famously small engineering team by following close to this order: it scaled PostgreSQL vertically and with replicas first, moved static content to a CDN, cached aggressively with memcached, and only then introduced logical sharding with the shard identifier embedded in object ids. The published accounts consistently emphasize choosing boring, well-understood technology and deferring each step until a measurement demanded it.
- Shopify has published extensively about its pods architecture, in which the platform is divided into isolated units each containing its own database and supporting services with shops assigned to pods, chosen specifically so that a failure or a hot shop is contained within one pod rather than affecting the whole platform, and so that capacity can be added by adding pods. It is a production example of shard-level isolation being adopted for blast radius reasons alongside capacity ones, which is the argument this course makes for treating shards as independent failure domains.
Review Questions & Answers
This is a self-directed synthesis exercise, so there is no single correct answer, and the value is in the discipline of naming the measured constraint before naming a remedy. The common outcome is discovering that a system has adopted a later-stage technique, most often sharding or an elaborate caching layer, without having exhausted an earlier one, or that nobody can state the current bottleneck in measured terms at all, which means any architectural change would be a guess. The useful output is a short list: what resource is closest to saturation, what the cheapest remaining remedy for it is, and what specific measurement would indicate that the next stage has genuinely become necessary.
Because each earlier step is strictly cheaper in complexity and often removes the need for the later one entirely. A CDN and a cache require no change to the data model, no key decisions, and no query restrictions, and they frequently remove the large majority of read load, which can defer replicas indefinitely. Read replicas require no data model change either and solve read throughput completely, at the cost of replication lag and read-your-writes handling, which is a bounded and well-understood problem. Sharding is the only step that permanently constrains the data model, makes cross-shard queries expensive, complicates transactions, and multiplies every future schema change, and it solves a problem, write throughput and dataset size, that the earlier steps genuinely cannot address. Taking it early means paying its permanent costs to solve a problem a cache would have solved, which is the most common and most expensive ordering mistake in the sequence.
A moving tail over a stationary median means a subset of requests is slow, not the system, and the per-shard and per-key views confirm it is one shard and a handful of keys. Doubling the shard count redistributes the key space, but each hot key still hashes to exactly one shard, so the hot keys arrive in their new homes exactly as hot as before. The fleet cost doubles, the cross-shard query cost rises because fan-out now touches twice as many shards, and the actual problem is untouched. The correct sequence is to identify the specific hot keys by sampling, apply local in-process caching for the read-hot ones, which typically reduces shared store load by several orders of magnitude, add request coalescing on the miss path, apply per-key rate limits so one key cannot consume a shard capacity and harm its neighbours, and where the skew is tenant-shaped, move the largest tenants to dedicated logical shards through the directory layer. Only after the distribution has been addressed does adding aggregate capacity make sense, and by then it usually is not needed.
The logical shard indirection: hashing keys into a large fixed number of logical shards and mapping those to physical machines through a small directory, rather than hashing keys directly to machines. At the moment sharding is introduced, it is a handful of lines and one configuration table. Without it, every subsequent capacity change remaps a large fraction of all keys and requires a full global data reshuffle with no safe intermediate state, so growing the fleet becomes a project rather than an operation. With it, adding capacity means copying whole logical shards, verifying each one, and flipping its mapping entry, one shard at a time and reversibly. It is also what makes moving a single large tenant to dedicated capacity a configuration change at stage five rather than a redesign, which is exactly the hot spot mitigation the system will need. The general lesson is that the cheap thing to build early is the indirection that preserves future options, not the capacity itself.
Knowing the techniques is knowing what each one does once you have decided to apply it. Being able to scale a system is knowing which constraint is actually binding right now, which is a measurement question rather than a knowledge question, and then choosing the smallest change that relieves it. Nearly every technique in this course is correct in some context and actively harmful in another: caching helps enormously with a high hit rate and makes things slower with a low one, sharding is essential past a write ceiling and pure cost before it, affinity routing is the point of consistent hashing and the cause of hot keys, and load-aware balancing is an improvement over round robin and a mechanism for routing traffic into a failing server. The skill is holding the diagnostic order, measure, exhaust the cheaper remedy, change the smallest thing, verify the constraint moved, rather than reaching for whichever technique is most familiar or most recently read about.