Catalog My learning Blog
Scalability Patterns: Load Balancing, Caching, Sharding · Phase 4 · Lesson 20 · 11 min read

Celebrity Problem and Flash Sales: Fan-Out at Scale

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Flash Sale That Degrades Instead of Collapsing
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Compare fan-out on write and fan-out on read for social feed delivery, explain why extreme follower counts force a hybrid, and describe the queueing, inventory, and admission techniques that make a flash sale survivable rather than merely fast.

Concept Explanation

Social systems and commerce systems produce hot spots with the same underlying shape but opposite timing. A celebrity account is a persistent hot spot driven by a distribution that is stable over months; a flash sale is a transient hot spot driven by a scheduled moment where load rises by orders of magnitude in seconds and is concentrated on a single item. Both defeat systems that were sized from averages, and both are foreseeable, which means both should be designed for rather than discovered.

Feed delivery is the canonical celebrity problem. Fan-out on write means that when a user posts, the system immediately writes a reference into the precomputed timeline of every follower. Reads then become trivial, a single sequential lookup of an already-materialized list, which is exactly what you want for an operation performed far more often than writing. The cost is that a post by an account with ninety million followers becomes ninety million writes, which is not merely expensive but a latency and capacity event in its own right, and it is wasted for the large fraction of followers who are inactive and will never read it. Fan-out on read inverts this: writes are cheap, just one insert into the author own list, and a timeline is assembled at read time by querying the accounts a user follows and merging. That makes celebrity posts cheap to publish but makes every timeline read a fan-out whose latency is governed by the slowest source, which is the tail latency problem from the sharding phase appearing in a new context.

The resolution every large system converges on is a hybrid. Ordinary accounts, which is the overwhelming majority, use fan-out on write, because their follower counts are small and precomputed timelines make reads fast. Accounts above a follower threshold are excluded from fan-out, and their posts are merged in at read time from a small set of sources that is cheap to query and, critically, is heavily cached because the same celebrity posts are being merged into millions of timelines. Additional refinements follow the same logic: fan out only to recently active followers and let dormant users assemble on read when they return, prioritize the delivery queue so that the fan-out for an account with ten million followers does not delay ordinary posts, and treat the fan-out as an asynchronous job with explicit backpressure rather than something performed inline with the write.

Flash sales concentrate the same pressure into a scheduled instant, with the added difficulty that the hot key is write-hot and correctness-critical: inventory must not be oversold. The demand is also largely illegitimate in the sense that if a thousand units are available and a million people arrive, 99.9 percent of the traffic cannot succeed by definition, so the design goal is to reject that traffic as early and as cheaply as possible rather than to serve it. The techniques follow from that. Move the queue to the edge, so the vast majority of requests never reach application servers, using a virtual waiting room that admits a controlled rate of users. Make the inventory decrement the only serialized operation, using an atomic conditional decrement or a pre-generated pool of reservation tokens, and use key splitting from the previous lesson when the write rate on a single item exceeds what one key can absorb. Separate reservation from payment, since holding a short-lived reservation lets the slow payment path run without holding a lock on inventory. And pre-warm everything: caches, connection pools, autoscaling groups, and CDN content, because autoscaling reacts in minutes and a flash sale peaks in seconds, which makes reactive scaling structurally useless for this specific shape of load.

The final and most important point about both cases is honesty about degradation. In a flash sale, the correct behaviour when demand exceeds supply is to tell most users quickly and clearly that they are in a queue or that the item is gone, not to let them wait on a spinner while the system attempts an impossible task. In feed delivery, the correct behaviour under a celebrity spike is to serve a slightly stale timeline rather than to block on a fan-out. In both, the system that performs best is the one that decided in advance which promise it would break.

ApproachWrite CostRead CostFails WhenUsed For
Fan-out on writeOne write per followerOne sequential readFollower counts reach millionsThe vast majority of ordinary accounts
Fan-out on readOne writeA query per followed account, then a mergeUsers follow many accounts; tail latency dominatesCelebrity accounts only
HybridFan-out for ordinary, none for celebritiesRead precomputed, merge a few cached sourcesThe threshold is tuned badlyEssentially every large social system
Active-follower fan-outOne write per recently active followerPrecomputed for active, assembled for returning usersActivity data is inaccurateReducing wasted writes to dormant accounts

Worked Example: A Flash Sale That Degrades Instead of Collapsing

The sale below has one thousand units and expects roughly a million simultaneous attempts. The design works backwards from the observation that at most a thousand requests can succeed, so nearly all of the engineering effort goes into rejecting the rest cheaply.

Python
# Sale: 1,000 units at 12:00:00. Expected arrivals: ~1,000,000 in ~10s.
# Success ceiling: 0.1% of requests. The job is to say no, fast and early.

# --- Layer 1: the edge. Reject 97% before it reaches any server. ---
# A virtual waiting room at the CDN issues signed admission tokens at a
# controlled rate. No token, no origin request.
#   admit_rate = 3,000/sec   (3x the unit count, absorbing abandonment)
#   everyone else: a static queued page served entirely from the edge,
#   with an honest position estimate. Zero origin load.
# Arrivals reaching the origin: ~3,000/sec instead of ~1,000,000/sec.

# --- Layer 2: pre-generated reservation tokens. No contention at all. ---
# BEFORE the sale opens, materialize exactly 1,000 tokens in a queue.
# The "inventory decrement" becomes an atomic pop from a list, which is
# O(1), cannot oversell by construction, and needs no lock.
BEFORE:  tokens = queue.push_all([f"tok-{i}" for i in range(1000)])

def attempt_purchase(user, admission_token):
    verify_edge_token(admission_token)          # cheap, stateless
    if not rate_limit_ok(user):
        return TOO_MANY_REQUESTS
    tok = queue.pop()                           # atomic; returns None when empty
    if tok is None:
        return SOLD_OUT                         # definitive, immediate, cheap
    reservation.create(user, tok, ttl=600)      # 10 minutes to pay
    return RESERVED(tok)
# Once the 1,000th pop succeeds, every subsequent request is answered by
# a single failed pop. SOLD_OUT costs almost nothing to serve, which is
# what lets the system absorb the remaining 999,000 attempts.

# --- Layer 3: payment is NOT on the hot path ---
# The slow, third-party, failure-prone step runs against a reservation
# the user already holds. Nothing about payment touches inventory.
def complete_payment(reservation):
    if reservation.expired():  return EXPIRED   # token returns to the queue
    charge(reservation)                         # seconds, external, retryable
    fulfil(reservation)

# Expired reservations are recycled by a sweeper, so abandoned carts do
# not permanently consume stock:
def sweeper():
    for r in reservations.expired():
        queue.push(r.token)                     # back into circulation

# --- Layer 4: pre-warm, because autoscaling cannot help here ---
#   T-60m  scale to full projected capacity (autoscaling reacts in
#          MINUTES; this event peaks in SECONDS - reactive scaling is
#          structurally useless for this load shape)
#   T-30m  warm caches with product detail, images, terms; warm the CDN
#   T-15m  pre-open database connection pools to full size
#   T-5m   freeze deploys; confirm the waiting room is armed
#   T-0    open

# --- What the user sees, which is the actual deliverable ---
#   in the queue      : "You are number 48,210. Estimated wait 4 minutes."
#   admitted, got one : "Reserved. Complete payment within 10 minutes."
#   admitted, too late: "Sold out."  (immediately, not after 30 seconds)
# Every one of these is a fast, truthful answer. The failure mode this
# design eliminates is the spinner that resolves into a timeout.
A flash sale request funnel narrowing through four layers, an edge waiting room admitting a controlled rate, a rate limiter, an atomic reservation token pop that cannot oversell, and a separate asynchronous payment path, alongside a social fan-out diagram showing ordinary accounts writing into follower timelines while a celebrity account is merged in at read time.
A flash sale request funnel narrowing through four layers, an edge waiting room admitting a controlled rate, a rate limiter, an atomic reservation token pop that cannot oversell, and a separate asynchronous payment path, alongside a social fan-out diagram showing ordinary accounts writing into follower timelines while a celebrity account is merged in at read time.

Real-World Examples

  • Twitter engineers have described publicly, and Martin Kleppmann recounts in Designing Data-Intensive Applications, that the service began with fan-out on read, moved to fan-out on write for read performance, and then adopted a hybrid because accounts with tens of millions of followers made a single post a delivery event large enough to be its own operational problem, so celebrity posts are merged at read time while ordinary posts are still delivered on write.
  • The November 2022 Ticketmaster presale for the Taylor Swift Eras Tour is a well-documented instance of flash sale demand exceeding a system design envelope: the company publicly reported approximately 3.5 billion total system requests on the day, several times its previous peak, and cancelled the subsequent public sale. The incident drew congressional attention and is a widely cited example of scheduled, foreseeable demand concentrated on a single inventory item overwhelming a platform built for continuous rather than instantaneous load.
Key TakeawaysIn a flash sale with a thousand units and a million shoppers, 99.9 percent of requests cannot succeed no matter how much capacity you provision, so the engineering problem is rejecting them cheaply and honestly rather than serving them. Move the queue to the edge, make sold out the cheapest response in the system, and remember that autoscaling reacts in minutes while the event peaks in seconds, so everything must be pre-warmed.

Review Questions & Answers

Fan-out on write makes reads trivial by precomputing each follower timeline, which is the right trade because timelines are read far more often than posts are written, but a post from an account with tens of millions of followers becomes tens of millions of writes, which is an operational event on its own and is wasted on the many followers who are inactive. Fan-out on read makes writes trivial but turns every timeline read, the most frequent operation in the product, into a fan-out over every followed account whose latency is bounded by the slowest source. The hybrid optimizes for the actual distribution: the overwhelming majority of accounts have small follower counts, so fan-out on write is cheap for them and gives fast reads, while the tiny number of accounts with enormous followings are excluded and merged at read time. Because those few celebrity sources are shared across millions of timelines, their content caches extremely well, so the read-time merge touches a small, hot, cacheable set rather than an arbitrary fan-out.

Because of the mismatch between the timescale of the load and the timescale of the response. Autoscaling observes a metric, evaluates it over a window, launches instances, waits for them to boot, warm their caches, fill their connection pools, and pass health checks, which realistically takes several minutes end to end. A flash sale reaches its peak within seconds of the scheduled moment, so the scaling action completes long after the event that triggered it has already succeeded or failed. The same applies to every other lazily-warmed resource: cold caches mean the first wave of requests all miss and hit the database simultaneously, cold connection pools mean connection establishment becomes part of the critical path at exactly the wrong moment, and cold CDN entries mean the origin serves assets it should never see. For gradual growth, reactive mechanisms work fine because the load changes more slowly than the response. For a scheduled spike, capacity must be in place and warm before the event opens, which is possible precisely because the event is scheduled.

Because it removes a slow, externally-dependent operation from the contended path. If inventory is only decremented on successful payment, then the inventory record must be held, locked, or otherwise reserved for the entire duration of a payment authorization, which involves a third party, takes seconds, and fails unpredictably. Under flash sale concurrency that turns a microsecond-scale contended resource into a seconds-scale one, collapsing throughput on the single key that matters most. Separating them means the contended operation is a single atomic decrement or token pop that completes in microseconds and cannot oversell, after which the user holds a time-limited reservation and the payment proceeds without touching inventory at all. Failed or abandoned payments are handled by expiry and a sweeper that returns tokens to circulation, so stock is not permanently lost. The user experience benefit, knowing immediately whether you have secured an item, is real but is a consequence of the throughput decision rather than the motivation for it.

The fan-out work is being processed from a single undifferentiated queue, so one job requiring fifty million writes occupies the workers for a long time and every ordinary post queued behind it waits, even though each of those needs only a few hundred writes. The problem is head-of-line blocking caused by an enormous variance in job size with no scheduling to account for it. The fixes compose: first, exclude accounts above a follower threshold from fan-out entirely and merge their posts at read time, which removes the largest jobs from the system rather than scheduling them better. Second, for the large jobs that remain, split them into many bounded chunks so no single unit of work is large, and process them on a separate queue or worker pool from ordinary posts so the two cannot block each other. Third, fan out first to recently active followers and handle dormant ones lazily, which cuts the work substantially without affecting anyone who is actually looking at their timeline.