Catalog My learning Blog
Scalability Patterns: Load Balancing, Caching, Sharding · Phase 4 · Lesson 21 · 11 min read

Capacity Planning and Load Testing for Hot Spots

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Capacity Model That Accounts for Skew
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Build a capacity model from measured per-unit costs rather than from aggregate guesses, design load tests whose traffic distribution reflects real skew instead of uniform random keys, and use game days and chaos exercises to verify the degraded modes a system claims to have.

Concept Explanation

Capacity planning fails most often not because the arithmetic is hard but because the model is built on the wrong quantity. A plan expressed as total requests per second divided by requests per server implicitly assumes every request costs the same and every server receives an equal share, and the preceding lessons have established that neither is true. A model that survives contact with production is built from measured unit costs: how much CPU, memory, disk, and downstream work a specific request type consumes, multiplied by the expected mix, plus explicit headroom for the fact that the busiest shard will be several times the average. The single most useful number to carry through the model is the ratio of peak shard load to average shard load, because that is the factor by which a uniform model understates what the busiest machine will actually face.

Headroom has to be reasoned about rather than picked. It covers at least four distinct things: normal peak-to-average variation within a day or week, the load that redistributes onto surviving instances when one fails, the burst that arrives during a retry storm or a cache miss event, and the time it takes for any scaling response to become effective. A fleet running at seventy percent of capacity with three instances loses roughly a third of its capacity when one fails, so the survivors immediately need to serve above one hundred percent, which is how a single instance failure becomes an outage. The general rule is to size so that the fleet remains within capacity after losing the largest failure domain it must tolerate, typically one instance for a small fleet and one availability zone for a large one.

Load testing deserves the sharpest criticism, because the default way it is done actively hides hot spots. A test that generates uniformly random keys produces perfectly even distribution across shards and cache slots, exercises a hit rate that no real workload would produce, and proves that the system works under conditions that will never occur. Realistic load testing requires reproducing the actual key distribution, which in practice means sampling real production traffic or generating keys from a Zipf-like distribution with a measured exponent, and reproducing the real request mix rather than hammering one cheap endpoint. It also means testing with a realistic dataset size, since a database whose entire working set fits in memory in a test environment tells you nothing about production, and testing the cold cache case explicitly, because that is a state the system will certainly enter and the one it is least likely to survive.

Several test types answer different questions and are worth distinguishing. A load test confirms behaviour at expected peak. A stress test increases load until something breaks, which is how you learn the actual limit and, more importantly, how the system behaves as it approaches and exceeds it: does it degrade along a chosen boundary or collapse. A soak test runs at moderate load for hours or days and finds leaks, unbounded growth, and slow degradation that a short test cannot. A spike test applies a step change in load and verifies that the response, including any autoscaling, is fast enough to matter, which is usually how teams discover that reactive scaling cannot help with a sudden spike. And a failure test removes a component under load and confirms that the documented degraded mode is the one that actually happens.

That last point generalizes into the most valuable practice available: deliberately exercising failure. Every claim a system makes about degraded operation, that it serves stale data when the cache is down, that it fails closed when authorization is unreachable, that losing a shard affects only that shard, is a hypothesis until it has been tested. Game days, where a team schedules a failure and works through it, and automated chaos experiments, where failures are injected continuously in controlled conditions, are how those hypotheses become facts. The specific failures worth rehearsing for the material in this course are: a cache tier at a zero percent hit rate, a single shard unavailable, a hot key at ten times its normal rate, a replica lagging by minutes, and a retry storm. Each of those is a scenario this course has argued will eventually happen, and each has a prepared response that is worth much more when it has been practised.

Test TypeQuestion It AnswersWhat It MissesCommon Mistake
Load testDoes the system meet its targets at expected peak?Anything past the limit, and anything slow to developUniform key distribution that hides every hot spot
Stress testWhere is the actual limit and how does it fail there?Behaviour over long durationsStopping at the first error instead of observing the collapse
Soak testDoes anything leak, grow, or degrade over hours?Peak and failure behaviourRunning too short to reach the leak
Spike testCan the system absorb a step change in load?Sustained capacityRamping gradually, which is not a spike
Failure testDo the documented degraded modes actually occur?Performance limitsTesting failures the system already handles well

Worked Example: A Capacity Model That Accounts for Skew

The model below is built from measured unit costs and then corrected by the two factors that a naive calculation omits: shard skew and failure headroom. The difference between the naive answer and the corrected one is a factor of more than four.

Code
# --- Measured unit costs (from production, per request type) ---
#   type            share   CPU ms   db queries   cache ops
#   product_read    0.62      1.8        0.1          2.4
#   search          0.11     14.0        1.0          1.2
#   cart_write      0.19      4.2        2.0          1.0
#   checkout        0.08     22.0        6.0          3.0
#
# Weighted CPU per request:
#   0.62*1.8 + 0.11*14.0 + 0.19*4.2 + 0.08*22.0 = 5.22 ms

# --- Naive model (wrong, and this is the one usually built) ---
#   peak traffic      = 180,000 req/sec
#   CPU needed        = 180,000 * 5.22ms = 940 CPU-seconds/sec = 940 cores
#   cores per instance= 16
#   instances         = 940 / 16 = 59
# This number is wrong in two independent ways.

# --- Correction 1: shard skew ---
#   measured max_shard_load / avg_shard_load = 3.4   (from production)
# Capacity must be sized so the BUSIEST shard fits, not the average one,
# because the busiest shard is what saturates first and it is what users
# on that shard experience.
#   effective requirement = 940 * 3.4 = 3,196 core-equivalents at the peak
#   shard, OR provision per-shard and let the hot shard have more.
# In practice: size per shard, and give the top shards extra capacity
# rather than uniformly over-provisioning the whole fleet.

# --- Correction 2: failure headroom ---
# Requirement: survive losing one availability zone of three.
#   surviving fraction = 2/3
#   instances must be sized so 2/3 of them carry 100% of peak:
#   instances = 59 / (2/3) = 89

# --- Correction 3: utilization target ---
# Never plan to 100%. Queueing theory: as utilization approaches 1,
# latency approaches infinity. Target 60-70% at peak.
#   instances = 89 / 0.65 = 137

# --- Result ---
#   naive:     59 instances
#   corrected: 137 instances (before shard-level skew allocation)
# The naive model is off by 2.3x on headroom alone, and does not model
# the hot shard at all. A fleet built to the naive number does not fail
# gradually; it fails the first time an AZ blips during a peak.

# --- The load test that would validate this ---
# WRONG:  keys drawn uniformly at random from the id space
#         -> perfectly even shards, unrealistically high cache hit rate,
#            proves the system works under conditions that never occur
# RIGHT:  keys replayed from a production traffic sample, or generated
#         from Zipf(alpha) with alpha fitted to measured production skew
#         -> reproduces the hot shard, the real hit rate, and the tail
# ALSO:   run it against a production-sized dataset, and run it once with
#         a COLD cache, because that is the state you are least likely
#         to survive and the one you will certainly enter.
A capacity model built up in stages from measured per-request cost to a final instance count, with each correction factor for shard skew, failure headroom, and utilization target shown as a multiplier, alongside two load test key distributions contrasting uniform random keys with a measured production skew.
A capacity model built up in stages from measured per-request cost to a final instance count, with each correction factor for shard skew, failure headroom, and utilization target shown as a multiplier, alongside two load test key distributions contrasting uniform random keys with a measured production skew.

Real-World Examples

  • Netflix built and published Chaos Monkey and the broader Simian Army, which terminate production instances during business hours specifically so that failures happen when engineers are present and so that resilience claims are continuously verified rather than assumed. The practice grew into the discipline now called chaos engineering, whose central premise matches this lesson: a degraded mode that has never been exercised is a hypothesis, not a capability.
  • Amazon runs a well-documented annual pre-Prime-Day readiness process including load testing and game days, and AWS has published Well-Architected guidance recommending that teams regularly test their resilience mechanisms and perform game days on production-like systems, precisely because scheduled high-demand events reward preparation and punish reactive scaling. The published emphasis on rehearsing failure before the event rather than reacting during it is the same argument this lesson makes for flash sales.
Common PitfallA load test that generates uniformly random keys is a test of a system you do not operate. It produces perfectly even shard distribution and an unrealistically high cache hit rate, so it validates the system under the exact conditions that never occur and hides every hot spot you are trying to find. Replay real traffic, or generate keys from a skew distribution fitted to production measurements.

Review Questions & Answers

It assumes two things that are reliably false. First, that all requests cost the same, when in reality a search or checkout can consume ten times the CPU and downstream work of a cached read, so the same request rate can imply very different capacity depending on the mix, and the mix shifts over time and during incidents. Second, that load is evenly distributed, when real key distributions concentrate traffic so that the busiest shard commonly carries several times the average. Since it is the busiest shard that saturates first, and users whose data lives there experience that saturation, capacity has to be sized against the peak shard rather than the mean. On top of both, the naive model usually omits failure headroom and a utilization target, and since these multiply rather than add, a model missing all of them can be off by a factor of four or more in a way that is invisible until the first simultaneous peak and failure.

A uniform-random key generator spreads requests evenly across the key space by construction, so every shard receives the same rate, every cache slot receives the same rate, and the max-to-median shard ratio is one. The hot key and hot partition phenomena simply do not exist in the test, so no amount of load will reproduce them, and the test will report that the system scales linearly with added shards, which is exactly the conclusion that a hot key invalidates. The cache hit rate is also wrong, and usually wrong in the optimistic direction if the test key space is small enough to fit entirely in cache, or pessimistic if it is enormous, but in neither case does it match the reuse pattern of real traffic. The test therefore validates a system with a uniform workload, which nobody operates, and the first time real skew meets the provisioned capacity is in production.

Because they are independent multipliers that can and do coincide. Peak-to-average headroom covers the difference between typical and busiest-hour traffic, which is a property of demand. Failure headroom covers the load that shifts onto surviving instances when part of the fleet is lost, which is a property of the topology. If a fleet is sized so that it is at ninety percent of capacity at peak, and it must survive losing one of three availability zones, then at peak with a zone down the survivors are asked to serve one hundred thirty five percent of what they can do, and they do not; they queue, latency rises, clients retry, and the partial failure becomes a total one. Sizing must satisfy both conditions simultaneously, which means the fleet must remain within its utilization target while serving peak traffic on the surviving fraction of instances, and that composition is what makes the corrected number so much larger than the naive one.

Because a degraded mode is a claim about behaviour under conditions that have never occurred, and such claims are wrong at a high rate. The stale-serving path has a code branch nobody has run, the fail-closed handler was written before two refactors, the runbook references a dashboard that has since moved, and the assumption that losing a shard affects only that shard turns out to be false because a shared metadata service is involved. All of these are discovered instantly by a rehearsal and otherwise only during a real incident, at the worst possible moment and with the least available attention. For this course material specifically, the rehearsals worth running are: take the cache tier to a zero percent hit rate and confirm the origin degrades rather than collapses; make one shard unavailable and confirm the blast radius is limited to it; drive a single key to ten times its normal rate and confirm the per-key limits and coalescing engage; lag a replica by minutes and confirm read-your-writes handling still works; and induce a retry storm and confirm the circuit breakers and retry budgets actually cap the amplification.