Rate Limiting & Throttling

8 min read

In a Nutshell

Rate limiting caps how many requests a client can make in a given time window — "100 requests per minute per user." It protects your API from abuse, prevents any single client from monopolizing resources, defends against denial-of-service, and enforces fair usage and pricing tiers. Throttling is the closely-related act of slowing down or rejecting requests that exceed the limit. Without these controls, one misbehaving client (a runaway script, a scraper, an attacker) can overwhelm your system and degrade service for everyone. Rate limiting is a foundational reliability and fairness mechanism that every production API needs.

2D minimalistic diagram showing a stream of requests from a client hitting a rate limiter gate; requests within the limit pass through (green), while requests exceeding the limit are rejected with a "429 Too Many Requests" response (red), with a counter showing "requests used / limit" resetting each time window

How It Actually Works

Why Rate Limit?

Reason	What It Prevents
Abuse/DoS protection	A flood of requests overwhelming your system
Fair usage	One noisy client starving others of capacity
Cost control	Runaway usage driving up infrastructure/third-party costs
Tiered pricing	Enforcing free vs paid quota limits
Backend protection	Shielding databases/downstream services from overload

The Core Algorithms

Algorithm	How It Works	Trade-off
Fixed Window	Count requests per fixed interval (e.g., per minute)	Simple; allows bursts at window edges
Sliding Window Log	Track timestamps of each request in a rolling window	Accurate; more memory
Sliding Window Counter	Weighted blend of current + previous window	Good accuracy, low memory
Token Bucket	Tokens refill at a rate; each request spends one; bucket has a max	Allows controlled bursts; the popular default
Leaky Bucket	Requests queue and drain at a fixed rate	Smooths output to a constant rate

Token Bucket: The Workhorse

The most widely used because it permits bursts while enforcing an average rate:

Bucket holds up to N tokens; refills at R tokens/sec.
  Each request removes 1 token.
  Token available → allow.   No token → reject (429) or wait.

  ┌─ capacity N=10 (max burst) ─┐
  │ ●●●●●●●●●●                    │  ← refills at R=5/sec
  └──────────────────────────────┘
  A client can burst up to 10 immediately, then is limited to 5/sec.

This models real usage well: occasional bursts are fine, sustained excess is not.

The Fixed-Window Edge Problem

Limit: 100/minute (fixed window).
  Client sends 100 at 11:00:59  and 100 at 11:01:00
  → 200 requests in 1 SECOND, but "legal" (spans two windows). 💥
Sliding-window and token-bucket algorithms avoid this boundary burst.

Communicating Limits to Clients

Good rate limiting is transparent — tell clients where they stand via headers, and how long to wait when limited:

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 42
X-RateLimit-Reset: 1718000000

# When exceeded:
HTTP/1.1 429 Too Many Requests
Retry-After: 30                 ← tells the client exactly when to retry

Returning 429 with Retry-After lets well-behaved clients back off intelligently instead of hammering.

Where and How to Key the Limit

Key by: user ID, API key, IP address, or a combination. IP-only is weak (shared NATs, easy to rotate); prefer authenticated identity where possible.
Where: commonly at the API gateway or a dedicated middleware, so backends are protected before load reaches them.
Distributed state: in a multi-server deployment, the counter must be shared — typically in Redis (atomic INCR/token-bucket scripts) — so the limit is global, not per-server.

Throttling vs Rate Limiting vs Load Shedding

Rate limiting:  reject requests OVER a client's quota (fairness/abuse)
Throttling:     deliberately slow/delay requests (smooth the rate)
Load shedding:  drop LOW-PRIORITY requests when the SYSTEM is overloaded
                (see graceful-degradation.md — system-driven, not client-driven)

2D minimalistic diagram of the token bucket algorithm: a bucket with a fixed capacity being refilled by a steady drip of tokens from above at a constant rate; incoming requests each remove one token to pass through; when the bucket is empty, requests are rejected, illustrating burst allowance up to capacity and a sustained average rate

Seeing It in Action

Scenario: Distributed rate limiting for a public API with tiered plans.

Plans:  Free = 60 req/min,  Pro = 1000 req/min,  Enterprise = custom.
Deployment: many API servers behind a load balancer → the counter MUST
be shared, so a client can't get "60 per server" by hitting different nodes.

Redis-backed token bucket (atomic via a Lua script):
  key = "ratelimit:{api_key}"
  On each request:
    - refill tokens based on elapsed time × plan rate
    - if tokens >= 1: decrement, allow
    - else: reject with 429 + Retry-After

Enforced at the API gateway (before backends):
  request → gateway → check Redis bucket for this api_key
     within limit → forward to backend, return headers:
        X-RateLimit-Limit: 1000, X-RateLimit-Remaining: 812
     over limit   → 429 Too Many Requests, Retry-After: 8
                    (never touches the backend — protection + cost savings)

Layered limits (defense in depth):
  - Per API key   (business/plan enforcement)
  - Per IP        (block abusive sources even pre-auth)
  - Global        (protect the whole system from a thundering herd)

Abuse scenario handled:
  A buggy Free-tier client loops infinitely → hits 60/min → gets 429s →
  its Retry-After tells it to back off → it can't degrade service for
  Pro/Enterprise customers, and your backend never sees the flood.

Why the distributed, gateway-enforced design matters: in a multi-server API, a naive per-server counter lets a client multiply its effective limit by the number of servers — so the counter must live in shared state (Redis) with atomic operations to be correct. Enforcing at the gateway means abusive traffic is rejected before it reaches (and costs) your backends. Layering per-key, per-IP, and global limits provides defense in depth: plan enforcement for business fairness, IP limits against pre-auth abuse, and a global cap as a last line against system-wide overload. Combined with clear 429 + Retry-After responses, well-behaved clients self-regulate while abusers are contained — protecting reliability, fairness, and cost simultaneously.

Interview Questions

Q: Why do APIs need rate limiting? Hint: To prevent abuse/DoS (a flood overwhelming the system), ensure fair usage (stop one client starving others), control costs (cap runaway usage), enforce tiered pricing/quotas, and protect backends/downstream services from overload. Without it, a single misbehaving client — script, scraper, or attacker — can degrade service for everyone. It's a core reliability and fairness mechanism.
Q: Explain the token bucket algorithm and why it's popular. Hint: A bucket holds up to N tokens and refills at R tokens/sec; each request spends a token, and requests are rejected when the bucket is empty. It's popular because it allows controlled bursts (up to capacity N) while enforcing a sustained average rate (R) — matching real usage where occasional bursts are fine but sustained excess isn't. It's memory-efficient and easy to implement.
Q: What's the problem with fixed-window rate limiting? Hint: Boundary bursts: a client can send the full limit at the end of one window and again at the start of the next (e.g., 100 at 11:00:59 and 100 at 11:01:00 = 200 in ~1 second), all technically "legal." Sliding-window (log or counter) and token-bucket algorithms avoid this by considering a rolling window or continuous refill rather than discrete resets.
Q: How do you rate limit correctly across many API servers? Hint: Use shared, atomic counter state (typically Redis with atomic INCR or a token-bucket Lua script) so the limit is global, not per-server. A naive per-server counter lets a client multiply its effective limit by the number of servers by spreading requests across nodes. Enforce at the gateway/middleware before backends, keyed by user/API key.
Q: How should an API communicate rate limits to clients? Hint: Via response headers showing the limit, remaining quota, and reset time (X-RateLimit-Limit/Remaining/Reset), and when exceeded, return 429 Too Many Requests with a Retry-After header telling the client exactly when to retry. This transparency lets well-behaved clients self-regulate and back off intelligently instead of hammering, improving overall stability.

References

Cloudflare: What is rate limiting? — concepts and algorithms
Stripe: Rate limiters — token bucket in production with Redis
MDN: 429 Too Many Requests — the standard response

Dive Deeper

System Design: Designing a rate limiter (ByteByteGo) — algorithm comparison in depth
Figma: An alternative approach to rate limiting — real-world design
Redis rate limiting patterns — distributed counters and buckets