How to Fix Hot Keys: Splitting and Local Caching
On this page
Learning Objective
Apply the four families of hot key mitigation, key splitting or salting, key replication across nodes, local in-process caching, and read and write path separation, and choose among them based on whether the key is read-hot or write-hot and whether it tolerates staleness.
Concept Explanation
Since a hot key cannot be relieved by adding nodes, every mitigation works by changing one of the three things that make it hot: how many physical locations the key occupies, how many requests actually reach those locations, or how the work is structured. The right choice depends primarily on one distinction that is worth making first, because it eliminates half the options immediately: is the key hot for reads or for writes? Read-hot keys are the easier case, because reads can be served from any number of copies and the mitigations are mostly about making more copies closer to the reader. Write-hot keys are harder, because writes must be ordered and reconciled somewhere, so the mitigations are about decomposing the write into independent pieces or deferring and aggregating it.
Local in-process caching is the first thing to reach for on a read-hot key, and it is the most effective by a wide margin because it removes the network hop entirely. If a key is read a hundred thousand times a second across two hundred application instances, caching it locally for even two seconds reduces load on the shared store to two hundred reads every two seconds, a reduction of five orders of magnitude, while adding no infrastructure. The cost is bounded staleness and per-instance divergence, which for genuinely hot keys such as configuration, feature flags, popular product metadata, or a trending item is almost always acceptable. This should be the default response to a read-hot key, and the fact that it is often overlooked in favour of scaling the cache tier is one of the more expensive habits in the field.
Key replication makes N copies of the same value under different keys, such as appending a replica index, and has readers pick one at random. Load on any single node drops by a factor of N, reads remain single-key lookups with no fan-out, and the scheme is simple. It works well for values that are read constantly and written rarely, because every write must now update all N copies, and the copies can briefly disagree. Key splitting, or salting, is the write-side analogue: instead of one counter key, maintain N subkeys each receiving a share of the increments, so write contention on any one is divided by N. The read path then becomes a fan-out that sums the subkeys, which is the price paid, and it is a good trade precisely when writes vastly outnumber reads, which is the normal shape for counters, rate limiters, and metrics.
Structural changes are often better than any of the above. A write-hot counter can be replaced by an append-only event stream aggregated periodically, which removes contention entirely because appends to different positions do not conflict. A hot lock can frequently be eliminated by making the operation idempotent or commutative so that ordering stops mattering. A hot key used for coordination, a leader election record, a sequence generator, a global rate limit, can often be replaced by a partitioned scheme where each instance owns a local budget and reconciles periodically. And the simplest structural fix for a read-hot key is to stop storing it in the shared store at all: configuration that changes rarely can be pushed to instances rather than pulled by them, which turns a hundred thousand reads per second into a broadcast on change.
Two operational notes matter. First, mitigations should ideally be adaptive rather than static, because hot keys move: the trending item today is not the trending item tomorrow. A system that detects the current top-N keys by request rate and automatically promotes them to local caching or splitting handles a moving target that a hardcoded list cannot. Second, some hot keys are best handled by admitting less traffic rather than serving more of it. If one key is consuming a shard entire capacity, a per-key rate limit protects every other key on that shard, converting a whole-shard outage into a degraded experience for the one hot item. That is almost always the better outcome, and it is the reason per-key quotas exist in mature storage systems.
| Mitigation | Works For | Read Cost | Write Cost | Main Tradeoff |
|---|---|---|---|---|
| Local in-process cache | Read-hot keys | Near zero after the first read | Unchanged | Bounded staleness and per-instance divergence |
| Key replication (N copies) | Read-hot, write-rare keys | One lookup, to a random copy | N writes per update | Copies can briefly disagree |
| Key splitting / salting | Write-hot keys such as counters | Fan-out over N subkeys | Divided by N | Reads become a fan-out and a sum |
| Request coalescing | Read-hot keys with expensive misses | One origin read per instance | Unchanged | Waiters share the latency of the single load |
| Structural change | Contention and coordination keys | Varies | Contention removed rather than divided | A redesign, not a configuration change |
| Per-key rate limiting | Any key threatening its neighbours | Rejected above the cap | Rejected above the cap | The hot item degrades to protect everything else |
Worked Example: The Same Key, Read-Hot and Write-Hot
The example below takes one product during a flash sale and treats its two hot keys separately, because the inventory counter and the product detail have opposite read-write profiles and therefore need opposite mitigations.
# product:8821 during a flash sale.
# detail (title, images, price) : 94,000 reads/sec, ~0 writes
# inventory (units remaining) : 3,000 reads/sec, 40,000 writes/sec
# ============ READ-HOT: product:8821:detail ============
# Best fix first: local cache. 200 app instances, 2s TTL.
local = TTLCache(maxsize=10_000, ttl=2)
def get_detail(pid):
v = local.get(pid)
if v is not None:
return v # 0 network hops
v = single_flight(pid, lambda: shared_cache.get_or_load(pid))
local[pid] = v
return v
# Shared-store load: 200 instances / 2s = 100 reads/sec.
# Down from 94,000. A 940x reduction, no new infrastructure, and the
# only cost is that a price change takes up to 2 seconds to appear
# everywhere - which is fine, and is a product decision, not an accident.
# If 2s staleness were NOT acceptable, the next option is replication:
# keys product:8821:detail:r0 .. :r15, reader picks r[rand(16)]
# load per node divided by 16; every write must update all 16.
# ============ WRITE-HOT: product:8821:inventory ============
# Local caching is useless (writes), replication is worse (16x the
# writes). The write must be DECOMPOSED.
SPLITS = 64
def reserve_unit(pid):
s = hash(instance_id, request_id) % SPLITS
# Each split holds its own slice of the inventory. Contention on any
# one key is 40,000 / 64 = 625 writes/sec, which is unremarkable.
if store.decr_if_positive(f"inv:{pid}:s{s}") is not None:
return RESERVED
# This split is exhausted; try a few others before declaring sold out.
for alt in random_sample(range(SPLITS), 4):
if store.decr_if_positive(f"inv:{pid}:s{alt}") is not None:
return RESERVED
return SOLD_OUT
def remaining(pid):
return sum(store.mget([f"inv:{pid}:s{i}" for i in range(SPLITS)]))
# Reads are now a 64-key fan-out. At 3,000 reads/sec that is fine;
# it would NOT be fine if the ratio were reversed.
# What splitting costs, stated honestly:
# - "sold out" becomes approximate at the boundary: a split can be empty
# while others have stock, which is why the retry loop above exists
# - remaining() is a sum over 64 keys, not a single read
# - rebalancing stock between splits needs a background job
# What it buys: the write path stops being a single-key bottleneck, which
# no amount of added hardware could have achieved.
# ============ AND: protect the neighbours ============
# Regardless of the above, cap what one key may consume so it cannot
# take down every other key on its shard.
per_key_limit("product:8821:*", max_ops_per_sec=120_000)
# Above the cap, requests for THIS key are shed. Every other product on
# shard 07 keeps working. Degrading one item beats losing a shard.Real-World Examples
- Amazon documents key sharding with a random or calculated suffix as the standard remediation for hot partition keys in DynamoDB, describing both a random suffix for write-heavy workloads and a calculated suffix derived from the item when reads must be able to locate a specific record without a full fan-out. The guidance is explicit that the read path becomes a multi-key query as a consequence, which is the trade this lesson describes.
- Netflix built and open-sourced EVCache as a caching layer that replicates data across zones and sits in front of its data stores, and has published that a local in-process caching tier in front of the remote cache was added specifically for very hot keys, because for those keys even a fast network round trip was a significant cost at Netflix request volumes. The layered approach, local cache in front of a replicated remote cache in front of the store, is the standard shape for read-hot data at large scale.
Review Questions & Answers
It is the only mitigation that eliminates the request rather than redistributing it. Every other approach still sends the read somewhere over a network; local caching answers it from memory in the same process, so the shared store sees, at most, one request per instance per TTL period regardless of how many times the key is read. With two hundred instances and a two second TTL, an arbitrarily large read rate collapses to a hundred requests per second at the store. The cost is precisely bounded staleness plus per-instance divergence: a change takes up to one TTL to be visible, and during that window different instances legitimately hold different values, so consecutive requests from one user can see the old value after having seen the new one. For configuration, flags, catalog metadata, and trending content, that is an acceptable and usually invisible cost. For a value where a user must observe their own write immediately, it is not, and one of the other mitigations applies instead.
Replication creates N copies of the value under distinct keys, which land on different nodes, so read load divides by N and each read remains a single lookup with no fan-out. That is close to ideal for a key read constantly and written rarely. For a write-hot key the arithmetic inverts: every update must now be applied to all N copies to keep them consistent, so write load is multiplied by N rather than divided, and the mitigation makes the exact problem it was applied to N times worse. It also introduces a window in which the copies disagree, which is tolerable for a slowly-changing value and unacceptable for one changing thousands of times per second. Write-hot keys need decomposition instead, splitting the single logical value into independent subvalues so that different writers touch different keys, which divides contention rather than multiplying it.
It gives up an exact, single-point view of the total. Once inventory is spread across sixty-four subkeys, no single read tells you the remaining count, so a total requires summing all of them, which is a fan-out and is only a momentary snapshot since other writers are decrementing concurrently. More importantly, individual splits can be exhausted while others still hold stock, so a naive implementation reports sold out while units remain. Real implementations handle this by retrying against a few other splits before concluding the item is unavailable, by running a background rebalancer that moves stock from full splits to empty ones, and by deliberately over-provisioning the last few units into a single reconciliation path where exact behaviour matters. The general principle is that splitting converts an exact serialized resource into an approximate distributed one, and the boundary conditions, the last few units, have to be handled explicitly rather than assumed away.
Because the alternative is usually worse for more people. A key consuming an entire shard capacity does not only degrade itself, it degrades every unrelated key co-located with it, so an uncapped hot key converts a problem affecting one item into an outage affecting a sixteenth of the dataset. A per-key cap contains the blast radius: requests for the hot key are shed above a threshold, and every other tenant, product, or user on that shard continues to be served normally. It is also immediately available during an incident, requiring only a configuration change, whereas the structural fixes are design work measured in days or weeks. Treating it as defeat assumes that serving all traffic is the goal, when the actual goal is maximizing the number of requests served correctly, and shedding the traffic that is crowding out everything else is straightforwardly the way to do that.