Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 2 · Lesson 20 · 5 min read

System Design: Design a Rate Limiter

On this page
  1. The problem: design a rate limiter for a public API
  2. The request path from DNS to service
  3. Designing the rate limiter: a token bucket in Redis
  4. Handling failures and scaling the rate limiter
  5. What a strong rate limiter design includes
  6. Check yourself

The problem: design a rate limiter for a public API

A company exposes a public REST API to thousands of partners. Each API key may make 1,000 requests per minute, with short bursts allowed. Over the limit, callers get a clear error. The API must stay fast and available, and partner traffic must never overload internal services. Design the full request path. Try it before reading the walkthrough.

The request path from DNS to service

Code
Partner  ->  DNS (Day 12)            api.company.example -> load balancer address
         ->  Load balancer (Day 19)  TLS, health-checked gateway instances, least connections
         ->  API gateway (Day 18)    auth by API key, rate limit, routing, logging
                 |  rate-limit check: Redis, atomic token bucket
         ->  Services                REST at the edge (Day 14), gRPC inside (Day 16)

Each hop has a job and a reason. DNS points at the load balancer, not at servers, so servers can change freely. The load balancer spreads connections over several gateway instances and drops unhealthy ones. The gateway applies the shared policies once, so services don't each reimplement them.

One request's path: DNS, load balancer, API gateway with a shared rate limiter in Redis, then routing to the right service.
One request's path: DNS, load balancer, API gateway with a shared rate limiter in Redis, then routing to the right service.

Designing the rate limiter: a token bucket in Redis

Algorithm. A token bucket per API key: capacity 100, refilled at about 16.7 tokens per second (1,000 per minute). Each request takes a token. Bursts up to 100 pass; the long-run rate is capped. A sliding window counter is a fine alternative with less burst allowance. Day 72 compares the algorithms in depth; here you only need the bucket.

Where. In the gateway, before any service is called. Rejected requests cost almost nothing.

Shared state. There are many gateway instances. If each counted in its own memory, a partner spread across 10 instances would get 10 times the limit. So counts live in Redis, keyed by API key.

This is exactly what happens with ASP.NET Core's built-in rate limiting middleware (.NET 7 and later): its limiters live in each process's memory. It's fine for protecting one instance from overload, but it isn't a per-customer quota across a fleet.

Atomicity. Reading the bucket, refilling it and taking a token must happen as one step, or two instances can both see one token left and both allow a request. A Lua script in Redis does it atomically; the System Design Interview course and the Redis course both walk through a tested script.

Responses. Over the limit: HTTP 429 Too Many Requests, with Retry-After and headers stating the limit and remaining quota, so well-behaved clients slow down instead of hammering.

Before the key is known. Limiting per API key only works after the key is checked, and checking it costs work. A flood of requests with invalid or missing keys must be stopped earlier: a coarse per-IP limit at the load balancer, WAF or CDN, before the gateway does any lookup.

Handling failures and scaling the rate limiter

  • Redis down. Decide in advance. Usually fail open: let requests through unlimited for a short time and alert, rather than turning a limiter problem into a full API outage. Run Redis with a replica and failover, and keep the call timeout to a few milliseconds.
  • Gateway instance down. Health checks remove it; others continue, because all state is in Redis.
  • Growth. One Redis key per active API key is tiny. At very high request rates, shard Redis by API key, or allow small local token allowances per instance that sync periodically, trading exactness for fewer Redis calls.

What a strong rate limiter design includes

  • The full path, with each component's job stated.
  • A named algorithm and why it fits bursty partner traffic.
  • Shared, atomic state, and why per-instance counters fail.
  • Proper 429 responses with retry information.
  • A stated fail-open or fail-closed choice.
  • No business logic in the gateway.

Check yourself

Each request is checked once at the edge, before it reaches any service, and the policy is implemented in one place instead of many.

The built-in limiters keep counts in each instance's memory, so the partner gets up to 1,000 per instance. Per-customer quotas need shared state, such as Redis.

A client's requests spread across instances, each allowing the full limit, so the effective limit is multiplied by the number of instances.

The bucket holds up to a maximum number of tokens and refills at a steady rate. Each request takes a token, so a client can spend a full bucket in a quick burst, but over time it can't exceed the refill rate. Rejected requests get 429 Too Many Requests with a Retry-After header.

It depends on the risk. Failing open keeps the API available but removes protection, which may be fine for internal fairness limits. Failing closed protects the backend but turns a Redis outage into an API outage. Many designs fail open with a local, per-instance limit as a fallback.