Catalog My learning Blog
Scalability Patterns: Load Balancing, Caching, Sharding · Phase 2 · Lesson 10 · 11 min read

Multi-Layer Caching and Cache Coherence

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Computing Real Worst-Case Staleness and Ordering a Purge
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Map the full stack of caches a single request can traverse, from the browser through the CDN and application tiers to the database buffer pool, explain why the effective staleness of a response is the sum of the layers rather than the value configured at any one, and describe how to keep multiple layers coherent without invalidating all of them on every write.

Concept Explanation

A request in a modern system passes through more caches than most engineers can name without thinking. The browser holds an HTTP cache governed by response headers and, often, a service worker cache with its own rules. Between the browser and the origin sit one or more CDN tiers, frequently an edge tier close to the user backed by a shield or mid-tier that absorbs edge misses so the origin sees far fewer of them. At the origin there is typically a reverse proxy cache, then an in-process local cache inside each application instance, then a shared distributed cache such as a memcached or Redis tier. Behind that the database has its own query plan cache and, most importantly, a buffer pool that keeps hot pages in memory. Each of these is doing useful work, and each is an independent copy of data that can be wrong.

The consequence that surprises teams is that staleness composes additively. If the CDN holds a response for 60 seconds, the application cache holds the underlying object for 300 seconds, and the browser was allowed to cache for 60 seconds, then a user can observe data that is over seven minutes old even though no single configured value exceeds five minutes. Worse, the layers are populated at independent times, so the worst case is not merely the sum in theory, it is routinely observed in practice. The practical discipline is to write down the layer stack for each important response type and compute the actual worst-case age, rather than reasoning about whichever layer happens to be in front of you. A related trap is that only the layers you control can be invalidated: you can purge a CDN, but you cannot reach into a browser cache that you already told to hold something for a day, which is why long client-side TTLs should be reserved for immutable, content-addressed URLs.

The single most useful structural technique is to cache different things at different layers rather than the same thing repeatedly. Caching a fully rendered response at the CDN and also caching the objects that response was built from at the application tier is coherent and useful, because the two layers absorb different traffic: the CDN serves anonymous repeat traffic without touching the origin at all, and the application cache serves the personalized or dynamic assembly path where the CDN cannot help. Caching the same rendered response at four consecutive layers, by contrast, multiplies the invalidation surface without multiplying the benefit, since after the first layer hits, the ones behind it are never consulted anyway. Layer separation by content type matters too: immutable versioned assets should be cached aggressively and forever at every layer, because their URL changes when their content does, while anything personalized should generally not be cached at a shared layer at all.

Local in-process caches deserve specific attention because they are the layer most likely to be coherent-by-accident rather than by design. They are extremely fast, with no network hop and no serialization, and they absorb the hottest keys so effectively that they are often the difference between a distributed cache tier being comfortable and being saturated. But each application instance has its own copy, there is no way to delete a key across all of them without a broadcast, and instances will legitimately disagree during the propagation window. The standard pattern, often called a near cache, is to give local entries a deliberately short TTL measured in seconds, which bounds divergence without needing any invalidation mechanism, and to add a pub-sub invalidation broadcast only for the specific keys where seconds of divergence is genuinely unacceptable. The failure mode to avoid is a long-lived local cache with no invalidation path, because then a single instance can serve stale data indefinitely while its peers are correct, which produces the maddening symptom of a page that shows the right value on some refreshes and the wrong value on others.

Coherence across layers is best achieved by propagating one invalidation signal outward rather than by having every writer know every layer. The event-driven pattern from the previous lesson generalizes cleanly: a change event is emitted once, and each layer subscribes with its own handler, the distributed cache bumping a version, the local caches receiving a broadcast, and the CDN receiving a purge-by-tag call. Ordering matters and runs from the back forward: invalidate the innermost layer first, because if you purge the CDN before clearing the application cache, the CDN immediately refills from the origin with the stale value it was supposed to be dropping and you have accomplished nothing except a brief load spike.

LayerScopeTypical TTLCan You Invalidate ItWhat It Should Hold
Browser / service workerOne user, one deviceSeconds to a yearNo, not once servedImmutable versioned assets only
CDN edgeOne region, all usersSeconds to hoursYes, by URL or surrogate keyAnonymous full responses and static assets
CDN shield / mid-tierAll edgesSimilar to edgeYesThe same, to protect the origin from edge misses
Reverse proxy at originOne datacenterSeconds to minutesYesFull responses, with stale-while-revalidate
Local in-process cacheOne instanceA few secondsOnly by broadcastThe very hottest keys, with short bounded divergence
Distributed cache tierAll instancesMinutes to hoursYes, directlyObjects and query results shared across the fleet
Database buffer poolOne database nodeManaged by the engineNot meaningfullyHot pages; tuned by sizing, not by application code

Worked Example: Computing Real Worst-Case Staleness and Ordering a Purge

The first half below is the arithmetic teams usually skip; the second half is the ordering rule that makes a multi-layer purge actually work. Both are short, and both are the difference between a cache stack that behaves predictably and one that produces unreproducible bug reports.

Code
# --- Response: /api/products/8821, a price shown on a product page ---
#
# Layer                 configured TTL      contributes
# browser Cache-Control  max-age=60          up to  60s
# CDN edge               s-maxage=120        up to 120s
# CDN shield             s-maxage=120        up to 120s
# origin reverse proxy   proxy_cache 60s     up to  60s
# app local near-cache   5s                  up to   5s
# distributed cache      ttl=300             up to 300s
#                                            -----------
# WORST-CASE OBSERVED AGE                        665s  (~11 minutes)
#
# Nobody configured anything longer than 5 minutes. A user can still see
# an 11-minute-old price, because each layer was populated at a different
# moment and each one's clock runs independently.
#
# If the requirement is "price is never more than 2 minutes stale", the
# budget must be allocated across layers, not set per layer:
#   browser 0 (no-store for price) + edge 30 + shield 30 + proxy 20
#   + local 5 + distributed 30  = 115s worst case. Now it holds.

# --- Purge ordering: innermost first, always ---
on_price_change(pid):
    # 1. Source of truth is already updated.
    # 2. Distributed cache FIRST - it is what the origin will read.
    cache.incr(f"ver:product:{pid}")
    # 3. Local caches next, via broadcast; they read from the distributed
    #    tier on their next miss, which is now correct.
    pubsub.publish("invalidate", f"product:{pid}")
    # 4. Origin reverse proxy.
    proxy.purge(f"/api/products/{pid}")
    # 5. CDN LAST. Only now will a refill fetch the correct value.
    cdn.purge_by_tag(f"product-{pid}")

# Reversing steps 2 and 5 is the classic bug: purging the CDN first makes
# it immediately refetch from an origin that is still serving the stale
# value out of the distributed cache, so the CDN re-caches the old data
# and the purge appears to have done nothing at all.
A request traversing seven cache layers from browser to database buffer pool, with each layer TTL annotated and the worst-case staleness accumulating along the path, alongside a purge propagating in the opposite direction from the innermost layer outward.
A request traversing seven cache layers from browser to database buffer pool, with each layer TTL annotated and the worst-case staleness accumulating along the path, alongside a purge propagating in the opposite direction from the innermost layer outward.

Real-World Examples

  • Wikimedia operates a multi-tier caching stack in front of MediaWiki in which edge points of presence absorb most traffic and fall back to a smaller set of core datacenters rather than hitting application servers directly, and the foundation has published architecture documentation and cache hit statistics showing that the overwhelming majority of article reads never reach the application layer at all. The tiered structure exists specifically so that an edge miss is served by another cache rather than by the origin, which is the shield pattern described in this lesson.
  • Cloudflare and Fastly both document tiered caching as an explicit product feature, where edge locations fetch misses from a designated upper-tier cache rather than each going independently to the origin, and both publish the reasoning: with hundreds of edge locations, a single uncached object would otherwise produce hundreds of simultaneous origin requests for the same resource. The upper tier collapses those into one, which is request coalescing from the previous lesson applied at the geography level.
Key TakeawaysThe staleness a user experiences is the sum of every cache layer in the path, not the largest single TTL, and it is routinely several times what anyone configured. Write down the full layer stack for each important response, add the TTLs up, and allocate a total staleness budget across layers rather than setting each one independently and hoping.

Review Questions & Answers

The layers are independent and their timers are not synchronized, so their staleness composes. A browser can fetch a response one second before the CDN entry expires, and that browser copy then lives for its own full 60 seconds on top of the 59 seconds of age the CDN copy already had. Behind the CDN the same thing happened between the CDN and the reverse proxy, and again between the proxy and the distributed cache, and again between the distributed cache and the database read that populated it. With five layers each contributing up to 60 seconds, the worst-case observable age approaches five minutes even though no layer is misconfigured and no code is wrong. The fix is to treat total staleness as a budget to be divided among layers, and to set no-store or very short values at the layers you cannot invalidate.

An outer cache refills by requesting from the layer behind it, so its correctness depends on that layer already being correct at the moment of the refill. If the CDN is purged first, the very next request for that object misses at the edge and is forwarded to the origin, which still holds the stale value in its reverse proxy or distributed cache and serves it. The CDN faithfully caches that stale response for a fresh full TTL, so the purge has achieved nothing except causing an origin load spike, and the stale data now has a longer remaining life than before the purge. Going innermost first means that by the time each outer layer is dropped, everything behind it already returns the new value, so any refill is correct. The same principle explains why local caches must be invalidated before the proxy in front of them.

Every application instance has its own independent copy with no shared state, so there is no single place to delete a key and no inherent mechanism for one instance to learn that another has newer data. Since a load balancer distributes requests across instances, the same user refreshing a page can be served by instances holding different versions, producing a value that appears to flicker between old and new. The standard bound is a deliberately short TTL, on the order of a few seconds, which caps divergence without requiring any invalidation machinery at all and preserves most of the performance benefit, because the hottest keys are read many times per second and a five second TTL still yields an extremely high hit rate. Where seconds of divergence is genuinely unacceptable, a pub-sub broadcast is added for those specific keys, but applying that broadcast to everything reintroduces the coupling the local cache was meant to avoid.

Because caches behind a hit are never consulted. Once the outermost layer serves the response, every layer behind it contributes nothing to that request, so the marginal benefit of each additional identical copy is limited to the small fraction of traffic that misses the layer in front of it. Meanwhile the costs are fully additive: every layer is another copy to invalidate, another contribution to worst-case staleness, another place a purge can be forgotten or fail, and another consumer of memory. Caching different things instead makes each layer productive on a different traffic pattern, for instance a CDN holding complete anonymous responses while the application tier holds the individual objects used to assemble personalized responses that the CDN can never serve. The test is whether a layer absorbs traffic that the layer in front of it structurally cannot.