Request Coalescing, Admission Control, Backpressure
On this page
Learning Objective
Describe how request coalescing collapses duplicate concurrent work, explain why an unbounded queue is a latency trap rather than a buffer, and design an admission control and load shedding scheme that degrades a system along a chosen boundary instead of letting it collapse.
Concept Explanation
Under a hot spot, a large fraction of the concurrent work in a system is duplicated: many callers are asking for the same thing at the same time. Request coalescing, also called single flight or request collapsing, exploits this directly. When a request for a key arrives and an identical request is already in flight, the new caller is attached to the existing operation rather than starting a second one, and when the original completes, every attached caller receives the same result. The saving is proportional to the concurrency, so it is largest exactly when the system is under the most pressure, which is an unusually good property. It is also simple, requires no coordination across processes, and has no distributed failure modes, which makes it one of the highest value-to-complexity mitigations available.
Coalescing has boundaries worth being precise about. It only deduplicates within the process that implements it, so with a large fleet the origin still sees one request per instance, which is generally an enormous reduction and occasionally not enough, in which case a coalescing tier closer to the data or a distributed lock is added on top. It applies only to idempotent reads or to operations where sharing a result is semantically correct; coalescing two writes is usually wrong. And it couples the attached callers to the fate of the single in-flight operation, so if that operation is slow, everyone waits, and if it fails, everyone fails together, which makes a per-caller timeout independent of the shared operation timeout important.
Admission control is the broader discipline that coalescing belongs to, and its core premise is that every system has a maximum amount of work it can perform, and the only choice is whether that limit is enforced deliberately or discovered catastrophically. A system without admission control accepts every request, so as load rises past capacity the work in progress grows, per-request latency grows with it, clients time out, retries add more load, and throughput collapses while utilization stays at one hundred percent. Goodput, the rate of useful completed work, falls to near zero precisely when demand is highest. With admission control, excess requests are rejected quickly and cheaply, the admitted requests are served at normal latency, and goodput stays flat at capacity instead of collapsing. This is the difference between a system that degrades and one that fails.
Queues are where this goes wrong most often, because a queue looks like a solution. An unbounded queue does not add capacity; it converts an overload into a latency problem and then hides it. Little Law states that the average time in the system equals the average queue length divided by the throughput, so if arrivals exceed service rate the queue grows without bound and so does latency, and items are eventually served long after the client that requested them has given up. Work is being performed on results nobody will use, which is worse than not performing it. The corrections are to bound every queue explicitly so that a full queue rejects immediately, and to apply a deadline to every queued item so that work whose client has already timed out is discarded rather than executed. Discarding queued work that has expired is one of the fastest ways to recover a system in overload.
Beyond simple limits, two refinements matter. Adaptive concurrency limits, of the kind implemented in the gradient algorithms used by several service mesh proxies, infer capacity from observed latency rather than requiring a configured number, raising the limit while latency stays near its minimum and lowering it when latency rises, which handles capacity that varies with request mix and with the state of downstream systems. Prioritized shedding decides what to reject rather than shedding uniformly: user-facing requests over batch, paying tenants over free, first attempts over retries, and cheap requests over expensive ones. Shedding uniformly under load means a small fraction of every user requests fail, which can make the whole product appear broken, while shedding by priority keeps the most important paths fully functional. Finally, backpressure means propagating the limit upstream rather than absorbing it: a full component should signal its producers to slow down, so that the pressure is felt where the work originates instead of accumulating in buffers throughout the system.
| Technique | What It Prevents | Signal It Uses | Failure If Omitted |
|---|---|---|---|
| Request coalescing | Duplicate concurrent work on the same key | An identical in-flight request | N identical origin queries per hot key |
| Bounded queue | Latency growing without limit under overload | Queue depth against a fixed cap | Work completes after every client has given up |
| Request deadlines | Executing work nobody is waiting for | Time remaining in the request budget | Capacity spent on already-abandoned requests |
| Fixed concurrency limit | More in-flight work than the system can serve | A configured maximum | Throughput collapses while utilization reads 100 percent |
| Adaptive concurrency limit | The same, without a hand-tuned number | Latency relative to its observed minimum | A static limit is wrong whenever the mix changes |
| Prioritized shedding | Uniform degradation that breaks everything a little | Request class, tenant, retry status | All users see partial failure instead of some seeing none |
| Circuit breaker | Hammering a dependency that is already failing | Error rate over a window | A struggling dependency is prevented from recovering |
Worked Example: Coalescing, Deadlines, and Shedding in One Request Path
The three mechanisms below compose, and each covers what the others cannot. Coalescing removes duplicate work, deadlines remove pointless work, and shedding removes excess work, in that order of preference.
# --- 1. COALESCING: one in-flight load per key, per process ---
inflight = {} # key -> future
def get(key, deadline):
v = cache.get(key)
if v is not None:
return v
fut = inflight.get(key)
if fut is None:
fut = inflight[key] = spawn(lambda: load_and_store(key))
fut.on_done(lambda: inflight.pop(key, None))
# Each caller waits with its OWN deadline, not the shared one. A slow
# load must not hold every attached caller past their own budget.
return fut.await_until(deadline)
# Effect at 94,000 req/sec for one key across 200 instances:
# without coalescing : 94,000 origin loads/sec
# with coalescing : 200 origin loads/sec (one per instance)
# --- 2. DEADLINES: never execute work whose client has left ---
# The deadline is set once at the edge and propagates with the request.
def handle(req):
deadline = now() + req.header("x-timeout-ms", default=2000)
...
def worker_loop(queue):
while True:
job = queue.pop()
if now() >= job.deadline:
metrics.incr("dropped.expired")
continue # the caller timed out already; doing this
# work helps nobody and costs capacity
job.run(budget=job.deadline - now())
# --- 3. ADMISSION CONTROL: bound concurrency, shed by priority ---
LIMIT = adaptive_limit() # inferred from latency vs its minimum
PRIORITY = {
"checkout": 0, # never shed while anything else can be
"product_page": 1,
"recommendations": 2,
"retry": 3, # shed retries BEFORE first attempts
"batch_export": 4, # shed first
}
def admit(req):
if inflight_count() < LIMIT:
return ACCEPT
# Over the limit: shed the lowest-priority work already, and reject
# anything below the current shed floor.
floor = current_shed_floor() # rises as pressure rises
if PRIORITY[req.klass] > floor:
return REJECT_503(retry_after=compute_backoff())
preempt_lowest_priority_inflight()
return ACCEPT
# --- Why this beats "just queue it" ---
# Little's Law: L = lambda * W, so W = L / lambda.
# capacity = 10,000 req/sec
# arrival rate = 14,000 req/sec
# unbounded queue -> grows 4,000/sec forever; after 60s the queue
# holds 240,000 items and W = 24 seconds. Every
# client timed out at 2s. Goodput is ZERO while
# the machine is 100% busy.
# bounded at 2,000 -> W = 0.2s worst case; 10,000/sec served correctly,
# 4,000/sec rejected in microseconds with a clear
# 503 and a retry-after. Goodput is 10,000/sec.
# Same hardware. One collapses, one degrades.Real-World Examples
- Discord published that it placed intermediate data services written in Rust between its API and its database specifically to implement request coalescing, so that many concurrent requests for the same row are collapsed into a single database query, and reported that this substantially reduced the hot partition pressure that had previously caused latency spikes. It is a direct production instance of coalescing being adopted as the primary defence against duplicated concurrent work on hot data.
- Netflix open-sourced its concurrency-limits library implementing adaptive limits inspired by TCP congestion control, which infer a service capacity from observed latency rather than requiring a configured thread or connection count, and the Envoy proxy implements a comparable adaptive concurrency filter. Both exist because static limits are wrong whenever request mix or downstream health changes, which is continuously, and both are explicitly framed as protecting goodput under overload rather than maximizing accepted requests.
Review Questions & Answers
Because the number of requests it can collapse is the number that are concurrently in flight for the same key, and that number rises with load. At low traffic a key might be requested once every few seconds, so there is rarely anything in flight to attach to and coalescing does essentially nothing. During a hot spot, hundreds or thousands of requests for the same key are in flight simultaneously, and all but one of them are eliminated. The mechanism therefore scales its own protection with the severity of the event, which is the opposite of most capacity measures, where the margin is fixed and gets consumed as load grows. It is also why coalescing is worth implementing before there is any observed problem: it costs nearly nothing in normal operation and is at full strength exactly when needed.
Little Law states that the average number of items in a system equals the arrival rate multiplied by the average time each spends there, which rearranges to say that wait time equals queue length divided by throughput. If arrivals exceed the service rate, the queue length grows without bound, and therefore so does wait time. Items are eventually processed long after the clients that submitted them have timed out and probably retried, so the system spends all its capacity producing results nobody receives while its utilization reads one hundred percent. The first change is to bound the queue, so that once it is full new arrivals are rejected immediately and cheaply, which caps wait time at queue size divided by throughput and returns a fast error the client can act on. The second is to attach a deadline to every item and check it on dequeue, discarding work whose client has already given up, which reclaims the capacity being wasted on abandoned requests and is often the fastest way to pull a system out of overload.
An adaptive limit infers capacity from observed latency, typically by tracking the minimum latency seen under light load and treating any sustained increase above it as evidence of queuing, then raising the concurrency limit while latency stays near the minimum and lowering it when it rises. A static limit assumes capacity is a constant, which it is not: the cost of a request varies with its type, the same service is slower when a downstream dependency degrades, garbage collection and cache state change throughput moment to moment, and a deployment can alter the per-request cost overnight. A static limit set high enough for the cheapest workload will admit far too much of the expensive one, and a limit set safely for the worst case leaves most of the machine idle most of the time. The adaptive version tracks the real boundary continuously without anyone maintaining a number that was only correct on the day it was measured.
Because the perceived outcome depends on which requests fail, not how many. Uniform shedding at twenty percent means roughly one request in five fails for everyone, and since a single page view typically issues several requests, almost every user experiences at least one failure, so the entire product appears broken to essentially the whole user base. Prioritized shedding rejects the same volume but concentrates it where it matters least, dropping batch exports, analytics beacons, recommendation panels, and retries while keeping checkout, authentication, and core reads fully served. Most users then see a complete, working experience with a missing recommendation strip, and the revenue-critical paths do not degrade at all. Shedding retries before first attempts is especially valuable because retries are the traffic most likely to be amplifying the overload in the first place, so rejecting them attacks the cause rather than just the symptom.