Design a Rate Limiter and API Gateway in .NET
On this page
Learning Objective
Apply the system design framework to a rate limiter, compare token bucket, fixed window and sliding window algorithms, implement an atomic distributed limiter with Redis, decide where limits live and how they fail, and identify clients correctly behind proxies.
Concept Explanation
Following the framework from Day 91, start with requirements. A rate limiter restricts how much a client can do in a period, to protect the service from overload and abuse, to share capacity fairly, and to enforce plans. Functional requirements: limit by user, API key, or IP address; different limits per endpoint or plan; return a clear response with how long to wait. Non-functional requirements: add very little latency, work across many server instances, stay correct under concurrency, and decide explicitly what happens if the limiter's own storage fails. Estimate the scale: a million active clients with one counter each is a small amount of memory, but a hundred thousand requests per second means the limiter is on the hot path of every call and must be a single fast round trip.
The algorithm decides the behavior. A fixed window counter counts requests in each clock window, is simple and cheap, and permits a burst of double the limit across a window boundary. A sliding window log stores each request time and is exact but costs memory, while a sliding window counter approximates it by weighting the previous window. The token bucket adds tokens at a steady rate up to a capacity and spends one per request: it allows short bursts up to the capacity while enforcing the average rate, which is what most APIs want, and it is what Stripe describes using. A leaky bucket smooths output to a constant rate and suits protecting a downstream that cannot absorb bursts. Also consider concurrency limits, how many requests a client may have in flight, and load shedding, which drops low-priority work when the whole system is stressed.
Distributed correctness is the hard part. If each app instance keeps its own counter, the effective limit multiplies by the number of instances. A shared store such as Redis fixes that, but a read-modify-write from the application races, so the check and update must be atomic, done in a Lua script that Redis executes as a single operation. Decide the failure mode: if Redis is unreachable, failing open keeps the service available but unprotected, and failing closed protects the service but rejects legitimate traffic, so the answer differs by endpoint. Login and expensive endpoints often warrant closed or a local fallback, and cheap reads open. Keep counters keyed with an expiry so they clean themselves up, and watch for hot keys where one client generates enormous traffic.
Identifying the client correctly is a security matter. Limiting by IP address is fragile: many users share an address behind a corporate network, and attackers rotate addresses. The X-Forwarded-For header is set by clients unless a trusted proxy overwrites it, so an attacker can send a fake value to evade or to frame someone else, and the application must trust only the headers set by known proxies, configured through the forwarded headers middleware. Prefer limiting by authenticated identity, with IP-based limits as a coarse layer for anonymous traffic, and limit sensitive actions by several keys at once, such as per account and per IP for login, as Day 28 discussed. A rate limiter is not a complete denial of service defense; volumetric attacks need edge protection such as a CDN and web application firewall, covered on Day 135. Return 429 with Retry-After, and make well-behaved clients back off with jitter.
| Algorithm | Behavior | Trade-off |
|---|---|---|
| Fixed window | Count per clock window | Simple, allows a burst of twice the limit at boundaries |
| Sliding window log | Exact count over the last period | Memory grows with request volume |
| Sliding window counter | Weighted blend of two windows | Approximate, small memory |
| Token bucket | Refill steadily, spend per request | Allows bursts, enforces average rate |
| Leaky bucket | Constant outflow rate | Smooths traffic, adds queueing delay |
| Concurrency limit | Cap on in-flight requests | Protects against slow, expensive calls |
Worked Example: An Atomic Token Bucket in Redis, Wired into ASP.NET Core
The Lua script performs the check and update in one step, and the middleware applies it with a failure policy and trusted client identification.
-- KEYS[1] = bucket key. ARGV: capacity, refill_per_sec, now_ms, cost
-- Returns { allowed (1 or 0), tokens_remaining, retry_after_ms }
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local data = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(data[1]) or capacity
local ts = tonumber(data[2]) or now
tokens = math.min(capacity, tokens + ((now - ts) / 1000.0) * rate) -- refill for the elapsed time
local allowed = 0
local retry = 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
else
retry = math.ceil(((cost - tokens) / rate) * 1000)
end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil((capacity / rate) * 1000) * 2) -- idle buckets disappear
return { allowed, math.floor(tokens), retry }public sealed class RedisTokenBucket(IConnectionMultiplexer redis, ILogger<RedisTokenBucket> log)
{
private static readonly LuaScript Script = LuaScript.Prepare(File.ReadAllText("token-bucket.lua").Replace("KEYS[1]", "@key")
.Replace("ARGV[1]", "@cap").Replace("ARGV[2]", "@rate").Replace("ARGV[3]", "@now").Replace("ARGV[4]", "@cost"));
public async Task<LimitResult> TryAsync(string key, int capacity, double perSecond, int cost, bool failOpen)
{
try
{
var r = (RedisResult[])(await redis.GetDatabase().ScriptEvaluateAsync(Script,
new { key = (RedisKey)key, cap = capacity, rate = perSecond, now = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(), cost }))!;
return new LimitResult((int)r[0] == 1, (int)r[1], TimeSpan.FromMilliseconds((long)r[2]));
}
catch (RedisException ex)
{
log.LogError(ex, "Rate limiter store unavailable, failing {Mode}", failOpen ? "open" : "closed");
return failOpen ? LimitResult.Allowed : LimitResult.Denied(TimeSpan.FromSeconds(5)); // a deliberate, per-endpoint choice
}
}
}
// Middleware: identify the caller, apply the policy, and speak the standard language of limits.
app.Use(async (ctx, next) =>
{
var policy = ctx.GetEndpoint()?.Metadata.GetMetadata<LimitPolicy>() ?? LimitPolicy.Default;
var caller = ctx.User.Identity?.IsAuthenticated == true ? $"u:{ctx.User.GetUserId()}" : $"ip:{ctx.Connection.RemoteIpAddress}"; // RemoteIpAddress is corrected by trusted forwarded headers only
var res = await limiter.TryAsync($"rl:{policy.Name}:{caller}", policy.Capacity, policy.PerSecond, policy.Cost, policy.FailOpen);
ctx.Response.Headers["RateLimit-Remaining"] = res.Remaining.ToString();
if (!res.Allowed)
{
ctx.Response.StatusCode = StatusCodes.Status429TooManyRequests;
ctx.Response.Headers.RetryAfter = Math.Ceiling(res.RetryAfter.TotalSeconds).ToString();
return;
}
await next();
});
// Trust forwarded headers only from the known proxy, or attackers set their own X-Forwarded-For.
builder.Services.Configure<ForwardedHeadersOptions>(o =>
{
o.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
o.KnownProxies.Add(IPAddress.Parse("10.0.0.4")); // the load balancer, never "any"
o.ForwardLimit = 1;
});Design review checklist for the limiter
Where: edge (cheap, coarse, by IP) -> gateway (per key and plan) -> service (per user, per expensive operation)
Keys: account or key first, IP as a fallback, login by account AND IP
Failure: login and payment endpoints fail closed with a small local limiter as backup, cheap reads fail open
Client: 429 plus Retry-After, clients back off with jitter, never retry instantly
Ops: dashboards for rejections per key, alerts on a single key dominating traffic, ability to raise a limit without a deployTest the limiter under concurrency, in the same way Day 112 tested coupons: fire a burst of parallel requests and assert that no more than the capacity succeed.
Real-World Examples
- Stripe published a 2017 article describing four kinds of limiter it runs: a request rate limiter using the token bucket algorithm, a concurrent requests limiter, and two load shedders that protect the fleet by dropping less critical traffic when capacity is scarce. It illustrates that a production system layers several controls, each protecting against a different failure.
- GitHub's REST API documents a limit of 5,000 requests per hour for authenticated requests, with headers reporting the remaining allowance and the reset time, and much lower limits for unauthenticated ones. It is a familiar example of tying limits to identity and of communicating them in headers so clients can adapt.
Review Questions & Answers
It enforces an average rate while allowing short bursts up to the bucket capacity, which matches real client behavior, and it avoids the fixed window's boundary problem where twice the limit can pass across two adjacent windows.
With many instances, two requests can read the same remaining count and both proceed, exceeding the limit. Running the logic as a Lua script makes Redis execute the check and the update as one indivisible operation.
Clients can send any value, so an attacker can evade limits by rotating fake addresses or cause another address to be limited. Only headers set by configured trusted proxies should be used to determine the client address.
Failing closed protects sensitive or expensive endpoints, such as login or payments, at the cost of rejecting legitimate users. Failing open keeps low-risk read endpoints available while the limiter is unavailable, and the choice should be explicit per endpoint.