Backpressure and Queue-Based Load Leveling
On this page
The burst that takes down a slow dependency
A report service can generate 10 PDF reports per second. Most of the day it gets 3. At 9:00 every Monday, 600 users click "export" within a minute. If each click calls the report service directly, it receives 10 times its capacity, response times climb to minutes, requests time out, users click again, and the service falls over. The average load is fine; the burst is the problem.
What is backpressure?
Backpressure is a system's way of saying "slow down" when it can't keep up, instead of silently accepting work it will fail to finish. Forms it takes:
- Reject quickly: return 429 or 503 with
Retry-After, so callers back off (Day 74). - Block the producer: a bounded queue or buffer that makes the sender wait when full. TCP does this at the network level: a receiver's window tells the sender how much more it can accept.
- Shed load: drop low-priority work first, such as analytics, to keep critical work flowing.
Without backpressure, overload turns into unbounded queues, memory exhaustion and timeouts everywhere, often hurting work that would otherwise have succeeded.
The queue-based load leveling pattern
Put a queue between the bursty producer and the steady consumer:
API -> enqueue "generate report 4417" -> return 202 Accepted + status URL
queue
worker(s) -> take jobs at 10/second -> store PDF -> notify userThe API answers immediately. The 600-request burst sits in the queue and drains in about a minute at the worker's steady rate. The report service never sees more than it can handle.
This works when the average arrival rate is below the consumer's capacity. If work arrives faster than it's processed for a long time, the queue only grows. Then you need more consumers or backpressure.
Sizing and bounding a queue
- Drain time: queue depth ÷ processing rate. 600 jobs at 10 per second is 60 seconds. If users accept a few minutes, that's fine.
- Bound the queue: set a maximum depth or message age. Beyond it, reject new work with a clear message rather than accepting jobs that will finish an hour late.
- Scale consumers on queue depth, not CPU: autoscalers such as KEDA scale workers by the number of waiting messages.
- Watch the age of the oldest message. It's the most direct signal that consumers are falling behind.
- Deduplicate jobs. A user who clicks "export" three times while waiting shouldn't get three reports. Key jobs by user and parameters, and return the existing job instead of queueing another.
Under overload, serve the newest first
A queue served in arrival order has a cruel property when it's deeply backed up: every request waits behind thousands of older ones, many of whose users have already given up. The system spends its capacity on work nobody is waiting for, and new requests time out too.
Facebook described switching to adaptive LIFO for exactly this: under normal load, serve in order; when the queue is backed up, serve the newest requests first, since those users are still there, and drop requests that have waited too long (Maurer, "Fail at Scale", 2015). For interactive traffic, a short queue with a deadline per request beats a long queue every time.
Telling the caller: 202 Accepted and job status
The 202 Accepted pattern returns a job id and a URL to check status:
POST /reports -> 202 Accepted, Location: /reports/jobs/4417
GET /reports/jobs/4417 -> {"status": "queued", "position": 212}
GET /reports/jobs/4417 -> {"status": "done", "download_url": "..."}Or notify the user by email or WebSocket when it's done. Either way, the user knows the request was accepted and isn't tempted to click again.
Check yourself
It stores the burst and lets the service work through it at 100 per second; the backlog clears within minutes, without overloading the service.
When the average arrival rate exceeds processing capacity. The queue grows without limit, so you need more consumers or must reject work.
An unbounded queue hides overload until it exhausts memory or delivers results far too late. Rejecting early tells callers to back off while the result still matters.
Bound the queue and drop requests that have already waited past their deadline. When backed up, serving the newest first helps users who are still waiting.
Backpressure slows producers down by signaling that the consumer is full, for example by blocking writes to a bounded queue or returning 429. Load shedding drops some work outright, ideally the least important or oldest, to keep serving the rest. Systems usually need both.