Retry with Exponential Backoff and Jitter
On this page
How naive retries cause a retry storm
A downstream service has a two-second hiccup. Every caller retries immediately, then again, and again, in tight loops. The service, already struggling, now receives three or four times its normal traffic, so it can't recover, and the two-second hiccup becomes a twenty-minute outage. Retries are necessary; done carelessly, they amplify failure.
Exponential backoff with jitter
Wait longer after each failure: 0.4 s, 0.8 s, 1.6 s, 3.2 s, capped at some maximum. That's exponential backoff. It gives the struggling service time.
On its own, backoff still synchronizes callers: thousands of clients that failed at the same moment all retry at 0.4 s, then all at 0.8 s, hitting the service in waves. Add jitter, randomness in the delay, to spread them out. The "full jitter" approach picks a random delay between zero and the backoff value; AWS's architecture blog analyzed the variants and found full jitter spreads load well (Brooker, 2015).
# backoff.py: retry with exponential backoff and full jitter
import random, time
RETRYABLE = {408, 429, 500, 502, 503, 504}
class HttpError(Exception):
def __init__(self, status): self.status = status
def call_with_retries(fn, max_attempts=5, base=0.2, cap=5.0):
for attempt in range(1, max_attempts + 1):
try:
return fn()
except HttpError as e:
if e.status not in RETRYABLE or attempt == max_attempts:
raise # 400, 401, 404...: retrying won't help
delay = random.uniform(0, min(cap, base * 2 ** attempt)) # full jitter
time.sleep(delay)Against a call that failed three times with 503 and then succeeded, it waited 0.09, 0.08 and 0.63 seconds, then returned. Against a 400 it raised immediately without retrying. The ceilings for each attempt are 0.4, 0.8, 1.6, 3.2 and 5 seconds; the actual waits are random below them.
Which errors to retry
| Retry | Don't retry |
|---|---|
| Timeouts and connection errors | 400 Bad Request: the request is wrong |
| 503 Service Unavailable, 502, 504 | 401, 403: credentials won't fix themselves |
429 Too Many Requests, honoring Retry-After | 404 Not Found |
| 500, sometimes, for idempotent requests | 409 Conflict, 422 validation errors |
Retry only idempotent operations, or make them idempotent first. A timed-out POST /payments may have succeeded; retrying without an idempotency key (Day 65) can charge twice.
Retry budgets and retrying at one layer
- Cap attempts (3 to 5) and the total time, which must fit inside the caller's own deadline (Day 75).
- Retry at one layer. If the browser retries 3 times, the gateway 3 times and the service 3 times, one failure can become 27 calls downstream.
- Use a retry budget: allow retries only up to, say, 10 percent of normal traffic. When the budget is spent, fail fast. Combine with a circuit breaker (Day 56), which stops retries altogether once a dependency is clearly down.
- Respect
Retry-Afterwhen the server sends it.
In .NET, Polly's retry strategy supports exponential backoff with jitter out of the box; Resilience4j and most cloud SDKs offer the same. Check the defaults of whatever wraps it: .NET's AddStandardResilienceHandler() retries every HTTP method, POST included, unless you turn that off (Day 56).
Retries aren't only for HTTP. Message consumers retry too, and an immediate redelivery loop against a failing database is the same storm. Use the broker's delayed redelivery, and move a message to the dead-letter queue after a few attempts (Day 42).
Check yourself
Without jitter, every client that failed at the same moment retries at the same moments, after exactly 1, 2 and 4 seconds, creating synchronized waves of load that can knock a recovering service down again. Random delays spread the retries across each interval, so load arrives smoothly.
No. A 400 means the request itself is invalid, so sending the same request again will fail the same way and only adds load. Retry transient failures such as timeouts, 503 and 429 (respecting Retry-After), and fix the request for 4xx errors.
Jitter. Every client that failed together retries together. Randomize each delay, for example uniformly between zero and the backoff value.
Up to 3 × 3 × 3 = 27 downstream calls for one user request, which turns a small outage into a flood. Retry at only one layer, usually the outermost or the one closest to the failing dependency, cap the attempts, and use a retry budget or circuit breaker so retries stop when failures are widespread.
A limit on retries as a share of normal traffic, for example retries may add at most 10 percent extra requests. When failures spike, the budget runs out and further requests fail fast instead of multiplying load on a dependency that's already struggling.