Catalog My learning Blog
Full-Stack Developer Course: React, .NET and SQL Server · Phase 7 · Lesson 69 · 8 min read

.NET Resilience with Polly: Retries and Circuit Breakers

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Typed Client with the Standard Handler and a Custom Pipeline
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Protect your API from failures in the services it calls by combining timeouts, retries with backoff and jitter, circuit breakers and concurrency limits using the standard resilience handler, avoid retry storms, and restrict outbound calls so resilience does not become a security hole.

Concept Explanation

Every dependency your API calls will eventually be slow or unavailable: the payment provider, another service, the email API, the database. Without protection, one slow dependency ties up your threads and connections waiting, requests pile up, and a failure in a peripheral system becomes an outage of your whole application. Resilience patterns limit this damage. A timeout puts an upper bound on how long you wait. A retry handles brief glitches by trying again. A circuit breaker stops calling a dependency that is clearly failing, giving it time to recover and letting your API fail fast. A bulkhead, or concurrency limit, caps how many calls to one dependency can be in flight, so it cannot consume every resource. Together they turn cascading failures into contained ones.

Polly is the .NET library for these patterns, created by Michael Wolfenden and now a .NET Foundation project. Its modern form, resilience pipelines, is integrated with HttpClient through the Microsoft.Extensions.Http.Resilience package, which provides a standard handler combining a rate limiter, total request timeout, retry, circuit breaker, and per-attempt timeout in a sensible order with good defaults. You attach it to a typed client in one line and tune what you need. There is also a hedging handler that sends a second request if the first is slow, useful for latency-sensitive reads against replicated services. The names of the ideas come from earlier systems, such as Netflix's Hystrix library from 2012, which popularized circuit breakers.

The details make the difference. Timeouts should nest: each attempt has its own limit, and an overall limit covers all attempts, and both must be shorter than the timeout of whoever is calling you, or you will keep working on a request the caller has already abandoned. Retry only what is safe. GET, PUT, and DELETE are idempotent, but a POST that creates something is not unless it carries an idempotency key. Use exponential backoff with jitter, as Day 57 explained, and cap total attempts, because retries multiply load: if three layers each retry three times, one failure can become 27 requests hitting an already struggling service, a retry storm. A circuit breaker opens when the failure ratio in a window exceeds a threshold, rejects calls immediately for a break period, then allows a trial call through to test recovery.

Resilience needs a plan for what the user sees when the dependency is down. A fallback might return cached data, a default, or a clear degraded-mode message such as receipts will arrive later, rather than a generic error, and the React side should be designed to cope, as Day 57 covered. Log and measure breaker state changes, retries, and timeouts, since they are the earliest indicator of an unhealthy dependency. And apply security thinking to the outbound side: a resilient client that retries requests to a user-supplied address is an amplifier for server-side request forgery, so outbound calls should go only to allowlisted hosts, follow no redirects by default, and carry no credentials for the wrong destination, as Day 50 described.

PatternProtects againstSetting to think about
TimeoutA dependency that hangsPer attempt and overall, shorter than the caller's
Retry with backoff and jitterTransient glitchesIdempotent requests only, small attempt count
Circuit breakerA dependency that is downFailure ratio, sampling window, break duration
Bulkhead or concurrency limitOne dependency consuming all capacityMaximum in-flight calls per dependency
HedgingOccasional slow responsesSafe reads only, adds load
FallbackTotal loss of the dependencyCached or degraded answer, honest to the user

Worked Example: A Typed Client with the Standard Handler and a Custom Pipeline

The first block configures the standard resilience handler for a payment provider client. The second builds a custom pipeline for a non-HTTP dependency, and the test simulates failures.

C#
// dotnet add package Microsoft.Extensions.Http.Resilience
builder.Services.AddHttpClient<PaymentClient>(c =>
{
    c.BaseAddress = new Uri("https://api.payments.example/");            // fixed, configured host: never user supplied
    c.Timeout = Timeout.InfiniteTimeSpan;                                // the resilience handler owns timeouts
})
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    AllowAutoRedirect = false,                                           // no surprise redirects to other hosts
    PooledConnectionLifetime = TimeSpan.FromMinutes(5),
})
.AddStandardResilienceHandler(o =>
{
    o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);             // whole call including retries
    o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(3);                  // each attempt

    o.Retry.MaxRetryAttempts = 2;
    o.Retry.BackoffType = DelayBackoffType.Exponential;
    o.Retry.UseJitter = true;
    o.Retry.ShouldHandle = args =>
    {
        var req = args.Outcome.Result?.RequestMessage;
        // A POST is retried only when it carries an idempotency key the provider honors.
        var safeToRepeat = req is null || req.Method != HttpMethod.Post || req.Headers.Contains("Idempotency-Key");
        var transient = args.Outcome.Exception is HttpRequestException
            || (args.Outcome.Result is { } r
                && ((int)r.StatusCode >= 500 || r.StatusCode is HttpStatusCode.RequestTimeout or HttpStatusCode.TooManyRequests));
        return ValueTask.FromResult(transient && safeToRepeat);
    };

    o.CircuitBreaker.FailureRatio = 0.5;                                 // open when half the calls fail
    o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
    o.CircuitBreaker.MinimumThroughput = 10;
    o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(20);

    o.RateLimiter.DefaultRateLimiterOptions.PermitLimit = 50;            // bulkhead: at most 50 concurrent calls
});

public sealed class PaymentClient(HttpClient http, ILogger<PaymentClient> log)
{
    public async Task<ChargeResult> ChargeAsync(Guid orderId, decimal amount, CancellationToken ct)
    {
        using var req = new HttpRequestMessage(HttpMethod.Post, "charges")
        {
            Content = JsonContent.Create(new { orderId, amount }),
        };
        req.Headers.Add("Idempotency-Key", orderId.ToString());          // makes a POST safe to retry if the provider honors it
        try
        {
            using var res = await http.SendAsync(req, ct);
            res.EnsureSuccessStatusCode();
            return (await res.Content.ReadFromJsonAsync<ChargeResult>(ct))!;
        }
        catch (BrokenCircuitException)
        {
            log.LogWarning("Payment circuit is open; failing fast for order {OrderId}", orderId);
            throw new DependencyUnavailableException("Payments are temporarily unavailable.");   // mapped to a 503 with Retry-After
        }
    }
}
C#
// A custom pipeline for anything else: a fallback that returns cached rates when the exchange-rate service is down.
builder.Services.AddResiliencePipeline<string, RateSnapshot>("rates", (b, ctx) =>
{
    b.AddFallback(new FallbackStrategyOptions<RateSnapshot>
    {
        ShouldHandle = new PredicateBuilder<RateSnapshot>().Handle<HttpRequestException>().Handle<TimeoutRejectedException>(),
        FallbackAction = _ => Outcome.FromResultAsValueTask(ctx.ServiceProvider.GetRequiredService<RateCache>().LastKnown()),
    });
    b.AddRetry(new RetryStrategyOptions<RateSnapshot> { MaxRetryAttempts = 2, BackoffType = DelayBackoffType.Exponential, UseJitter = true });
    b.AddTimeout(TimeSpan.FromSeconds(2));
});
C#
// Map the fail-fast condition to an honest HTTP response.
app.UseExceptionHandler(); // existing handler: DependencyUnavailableException -> 503 with Retry-After: 20 and a ProblemDetails body

// Test: simulate a failing dependency and assert that the breaker opens and the API fails fast, not slowly.
[Fact]
public async Task After_repeated_failures_the_client_fails_fast()
{
    _handler.RespondWith(HttpStatusCode.InternalServerError);
    for (var i = 0; i < 15; i++) await Record.ExceptionAsync(() => _client.ChargeAsync(Guid.NewGuid(), 10m, default));

    var sw = Stopwatch.StartNew();
    await Assert.ThrowsAsync<DependencyUnavailableException>(() => _client.ChargeAsync(Guid.NewGuid(), 10m, default));
    Assert.True(sw.ElapsedMilliseconds < 100);                            // rejected by the open circuit, no network call
}

Notice that the payment call is retried only because it carries an idempotency key and the provider honors it, and that the base address is fixed configuration, so the resilience machinery can never be aimed at an address a user supplied.

A call to the payment API passes a total timeout, retries with backoff and jitter for safe requests, a circuit breaker that fails fast when open, a concurrency limit and a per-attempt timeout. Outbound hosts are fixed and redirects are off.
A call to the payment API passes a total timeout, retries with backoff and jitter for safe requests, a circuit breaker that fails fast when open, a concurrency limit and a per-attempt timeout. Outbound hosts are fixed and redirects are off.

Real-World Examples

  • Netflix released the Hystrix library in 2012 to isolate calls between its many services with timeouts, thread pool bulkheads, and circuit breakers, after cascading failures where one slow dependency exhausted resources across the system. It entered maintenance mode in 2018 in favor of adaptive approaches, but its vocabulary and patterns became the industry standard, and Polly implements the same ideas for .NET.
  • AWS and other cloud providers document retry storms as a common amplifier of outages: when a service degrades, clients across many layers retry, multiplying the load and preventing recovery. Their guidance, which the standard resilience handler encodes, is to limit attempts, add jitter, use circuit breakers, and retry at only one layer of the call chain.
Key TakeawaysProtect every outbound dependency so one slow service cannot take yours down, starting with nested timeouts on each call. Add limited retries with backoff and jitter on idempotent calls, a circuit breaker to fail fast, and a concurrency limit. Retry at only one layer to avoid storms, provide honest fallbacks, monitor breaker events, and keep outbound hosts fixed and allowlisted so resilience code cannot be turned into an SSRF amplifier.

Review Questions & Answers

When a dependency struggles, clients at several layers each retry, multiplying the number of requests, so a degraded service receives far more load exactly when it can least handle it. Avoid it by keeping attempts low, using exponential backoff with jitter, retrying at only one layer, and adding circuit breakers so calls stop when the failure is clear.

When failures exceed a threshold it opens and rejects calls immediately, so your API stops wasting threads and time waiting on a dead dependency and can return a fast, controlled error. The dependency also gets relief from the constant traffic, giving it space to recover, and a trial call after the break period detects when it is healthy again.

If your work continues after the caller has given up, you keep spending resources on a result nobody will read, and the caller may retry, creating duplicate work. Nesting timeouts so inner limits are shorter than outer ones ensures work stops before the caller abandons it and the failure is reported meaningfully.

Retries, redirects, and long timeouts make the server more persistent about reaching whatever address it is given, which amplifies server-side request forgery against internal services and cloud metadata endpoints. Outbound clients should use fixed configured hosts or strict allowlists, disable automatic redirects, and never forward credentials to unintended destinations.