Celebrity Problem and Flash Sales: Fan-Out at Scale
On this page
Learning Objective
Compare fan-out on write and fan-out on read for social feed delivery, explain why extreme follower counts force a hybrid, and describe the queueing, inventory, and admission techniques that make a flash sale survivable rather than merely fast.
Concept Explanation
Social systems and commerce systems produce hot spots with the same underlying shape but opposite timing. A celebrity account is a persistent hot spot driven by a distribution that is stable over months; a flash sale is a transient hot spot driven by a scheduled moment where load rises by orders of magnitude in seconds and is concentrated on a single item. Both defeat systems that were sized from averages, and both are foreseeable, which means both should be designed for rather than discovered.
Feed delivery is the canonical celebrity problem. Fan-out on write means that when a user posts, the system immediately writes a reference into the precomputed timeline of every follower. Reads then become trivial, a single sequential lookup of an already-materialized list, which is exactly what you want for an operation performed far more often than writing. The cost is that a post by an account with ninety million followers becomes ninety million writes, which is not merely expensive but a latency and capacity event in its own right, and it is wasted for the large fraction of followers who are inactive and will never read it. Fan-out on read inverts this: writes are cheap, just one insert into the author own list, and a timeline is assembled at read time by querying the accounts a user follows and merging. That makes celebrity posts cheap to publish but makes every timeline read a fan-out whose latency is governed by the slowest source, which is the tail latency problem from the sharding phase appearing in a new context.
The resolution every large system converges on is a hybrid. Ordinary accounts, which is the overwhelming majority, use fan-out on write, because their follower counts are small and precomputed timelines make reads fast. Accounts above a follower threshold are excluded from fan-out, and their posts are merged in at read time from a small set of sources that is cheap to query and, critically, is heavily cached because the same celebrity posts are being merged into millions of timelines. Additional refinements follow the same logic: fan out only to recently active followers and let dormant users assemble on read when they return, prioritize the delivery queue so that the fan-out for an account with ten million followers does not delay ordinary posts, and treat the fan-out as an asynchronous job with explicit backpressure rather than something performed inline with the write.
Flash sales concentrate the same pressure into a scheduled instant, with the added difficulty that the hot key is write-hot and correctness-critical: inventory must not be oversold. The demand is also largely illegitimate in the sense that if a thousand units are available and a million people arrive, 99.9 percent of the traffic cannot succeed by definition, so the design goal is to reject that traffic as early and as cheaply as possible rather than to serve it. The techniques follow from that. Move the queue to the edge, so the vast majority of requests never reach application servers, using a virtual waiting room that admits a controlled rate of users. Make the inventory decrement the only serialized operation, using an atomic conditional decrement or a pre-generated pool of reservation tokens, and use key splitting from the previous lesson when the write rate on a single item exceeds what one key can absorb. Separate reservation from payment, since holding a short-lived reservation lets the slow payment path run without holding a lock on inventory. And pre-warm everything: caches, connection pools, autoscaling groups, and CDN content, because autoscaling reacts in minutes and a flash sale peaks in seconds, which makes reactive scaling structurally useless for this specific shape of load.
The final and most important point about both cases is honesty about degradation. In a flash sale, the correct behaviour when demand exceeds supply is to tell most users quickly and clearly that they are in a queue or that the item is gone, not to let them wait on a spinner while the system attempts an impossible task. In feed delivery, the correct behaviour under a celebrity spike is to serve a slightly stale timeline rather than to block on a fan-out. In both, the system that performs best is the one that decided in advance which promise it would break.
| Approach | Write Cost | Read Cost | Fails When | Used For |
|---|---|---|---|---|
| Fan-out on write | One write per follower | One sequential read | Follower counts reach millions | The vast majority of ordinary accounts |
| Fan-out on read | One write | A query per followed account, then a merge | Users follow many accounts; tail latency dominates | Celebrity accounts only |
| Hybrid | Fan-out for ordinary, none for celebrities | Read precomputed, merge a few cached sources | The threshold is tuned badly | Essentially every large social system |
| Active-follower fan-out | One write per recently active follower | Precomputed for active, assembled for returning users | Activity data is inaccurate | Reducing wasted writes to dormant accounts |
Worked Example: A Flash Sale That Degrades Instead of Collapsing
The sale below has one thousand units and expects roughly a million simultaneous attempts. The design works backwards from the observation that at most a thousand requests can succeed, so nearly all of the engineering effort goes into rejecting the rest cheaply.
# Sale: 1,000 units at 12:00:00. Expected arrivals: ~1,000,000 in ~10s.
# Success ceiling: 0.1% of requests. The job is to say no, fast and early.
# --- Layer 1: the edge. Reject 97% before it reaches any server. ---
# A virtual waiting room at the CDN issues signed admission tokens at a
# controlled rate. No token, no origin request.
# admit_rate = 3,000/sec (3x the unit count, absorbing abandonment)
# everyone else: a static queued page served entirely from the edge,
# with an honest position estimate. Zero origin load.
# Arrivals reaching the origin: ~3,000/sec instead of ~1,000,000/sec.
# --- Layer 2: pre-generated reservation tokens. No contention at all. ---
# BEFORE the sale opens, materialize exactly 1,000 tokens in a queue.
# The "inventory decrement" becomes an atomic pop from a list, which is
# O(1), cannot oversell by construction, and needs no lock.
BEFORE: tokens = queue.push_all([f"tok-{i}" for i in range(1000)])
def attempt_purchase(user, admission_token):
verify_edge_token(admission_token) # cheap, stateless
if not rate_limit_ok(user):
return TOO_MANY_REQUESTS
tok = queue.pop() # atomic; returns None when empty
if tok is None:
return SOLD_OUT # definitive, immediate, cheap
reservation.create(user, tok, ttl=600) # 10 minutes to pay
return RESERVED(tok)
# Once the 1,000th pop succeeds, every subsequent request is answered by
# a single failed pop. SOLD_OUT costs almost nothing to serve, which is
# what lets the system absorb the remaining 999,000 attempts.
# --- Layer 3: payment is NOT on the hot path ---
# The slow, third-party, failure-prone step runs against a reservation
# the user already holds. Nothing about payment touches inventory.
def complete_payment(reservation):
if reservation.expired(): return EXPIRED # token returns to the queue
charge(reservation) # seconds, external, retryable
fulfil(reservation)
# Expired reservations are recycled by a sweeper, so abandoned carts do
# not permanently consume stock:
def sweeper():
for r in reservations.expired():
queue.push(r.token) # back into circulation
# --- Layer 4: pre-warm, because autoscaling cannot help here ---
# T-60m scale to full projected capacity (autoscaling reacts in
# MINUTES; this event peaks in SECONDS - reactive scaling is
# structurally useless for this load shape)
# T-30m warm caches with product detail, images, terms; warm the CDN
# T-15m pre-open database connection pools to full size
# T-5m freeze deploys; confirm the waiting room is armed
# T-0 open
# --- What the user sees, which is the actual deliverable ---
# in the queue : "You are number 48,210. Estimated wait 4 minutes."
# admitted, got one : "Reserved. Complete payment within 10 minutes."
# admitted, too late: "Sold out." (immediately, not after 30 seconds)
# Every one of these is a fast, truthful answer. The failure mode this
# design eliminates is the spinner that resolves into a timeout.Real-World Examples
- Twitter engineers have described publicly, and Martin Kleppmann recounts in Designing Data-Intensive Applications, that the service began with fan-out on read, moved to fan-out on write for read performance, and then adopted a hybrid because accounts with tens of millions of followers made a single post a delivery event large enough to be its own operational problem, so celebrity posts are merged at read time while ordinary posts are still delivered on write.
- The November 2022 Ticketmaster presale for the Taylor Swift Eras Tour is a well-documented instance of flash sale demand exceeding a system design envelope: the company publicly reported approximately 3.5 billion total system requests on the day, several times its previous peak, and cancelled the subsequent public sale. The incident drew congressional attention and is a widely cited example of scheduled, foreseeable demand concentrated on a single inventory item overwhelming a platform built for continuous rather than instantaneous load.
Review Questions & Answers
Fan-out on write makes reads trivial by precomputing each follower timeline, which is the right trade because timelines are read far more often than posts are written, but a post from an account with tens of millions of followers becomes tens of millions of writes, which is an operational event on its own and is wasted on the many followers who are inactive. Fan-out on read makes writes trivial but turns every timeline read, the most frequent operation in the product, into a fan-out over every followed account whose latency is bounded by the slowest source. The hybrid optimizes for the actual distribution: the overwhelming majority of accounts have small follower counts, so fan-out on write is cheap for them and gives fast reads, while the tiny number of accounts with enormous followings are excluded and merged at read time. Because those few celebrity sources are shared across millions of timelines, their content caches extremely well, so the read-time merge touches a small, hot, cacheable set rather than an arbitrary fan-out.
Because of the mismatch between the timescale of the load and the timescale of the response. Autoscaling observes a metric, evaluates it over a window, launches instances, waits for them to boot, warm their caches, fill their connection pools, and pass health checks, which realistically takes several minutes end to end. A flash sale reaches its peak within seconds of the scheduled moment, so the scaling action completes long after the event that triggered it has already succeeded or failed. The same applies to every other lazily-warmed resource: cold caches mean the first wave of requests all miss and hit the database simultaneously, cold connection pools mean connection establishment becomes part of the critical path at exactly the wrong moment, and cold CDN entries mean the origin serves assets it should never see. For gradual growth, reactive mechanisms work fine because the load changes more slowly than the response. For a scheduled spike, capacity must be in place and warm before the event opens, which is possible precisely because the event is scheduled.
Because it removes a slow, externally-dependent operation from the contended path. If inventory is only decremented on successful payment, then the inventory record must be held, locked, or otherwise reserved for the entire duration of a payment authorization, which involves a third party, takes seconds, and fails unpredictably. Under flash sale concurrency that turns a microsecond-scale contended resource into a seconds-scale one, collapsing throughput on the single key that matters most. Separating them means the contended operation is a single atomic decrement or token pop that completes in microseconds and cannot oversell, after which the user holds a time-limited reservation and the payment proceeds without touching inventory at all. Failed or abandoned payments are handled by expiry and a sweeper that returns tokens to circulation, so stock is not permanently lost. The user experience benefit, knowing immediately whether you have secured an item, is real but is a consequence of the throughput decision rather than the motivation for it.
The fan-out work is being processed from a single undifferentiated queue, so one job requiring fifty million writes occupies the workers for a long time and every ordinary post queued behind it waits, even though each of those needs only a few hundred writes. The problem is head-of-line blocking caused by an enormous variance in job size with no scheduling to account for it. The fixes compose: first, exclude accounts above a follower threshold from fan-out entirely and merge their posts at read time, which removes the largest jobs from the system rather than scheduling them better. Second, for the large jobs that remain, split them into many bounded chunks so no single unit of work is large, and process them on a separate queue or worker pool from ordinary posts so the two cannot block each other. Third, fan out first to recently active followers and handle dormant ones lazily, which cuts the work substantially without affecting anyone who is actually looking at their timeline.