Rate Limiter Pattern

In a Nutshell

The rate limiter pattern controls the rate at which operations are allowed to proceed — capping how many requests, calls, or actions can happen in a given time window. It's a protective pattern that shows up on both sides of a system: as a server-side control to protect your service from being overwhelmed by clients (abuse, traffic spikes, runaway loops), and as a client-side control to keep your own outbound calls within a downstream service's limits (so you don't get throttled or banned). This topic covers rate limiting as an implementable pattern — the algorithms and where to apply them — complementing the API-focused treatment in Topic 11 — Rate Limiting & Throttling.

How It Actually Works

Two Directions of Rate Limiting

Direction	Purpose	Example
Server-side (inbound)	Protect your service from clients	"100 req/min per user" to prevent abuse/overload
Client-side (outbound)	Respect a downstream's limits	Throttle your calls to a third-party API to avoid being banned

The same algorithms apply to both; the direction just changes what you're protecting.

The Core Algorithms

Algorithm	How It Works	Characteristic
Token Bucket	Tokens refill at a rate; each request spends one; bucket caps burst	Allows bursts, enforces average rate — the popular default
Leaky Bucket	Requests queue and drain at a fixed rate	Smooths output to a constant rate
Fixed Window	Count requests per fixed interval	Simple; boundary-burst problem
Sliding Window	Rolling window count (log or weighted counter)	Accurate, avoids boundary bursts

Token Bucket — The Go-To

Because it permits controlled bursts while enforcing a sustained average rate, token bucket is the most common choice:

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity          # max burst
        self.tokens = capacity
        self.refill_rate = refill_rate    # tokens per second
        self.last = time.time()

    def allow(self, cost=1):
        now = time.time()
        # Refill based on elapsed time, capped at capacity
        self.tokens = min(self.capacity,
                          self.tokens + (now - self.last) * self.refill_rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True                   # allowed
        return False                      # rate limited

A client can burst up to capacity immediately, then is limited to refill_rate sustained — matching real usage where occasional bursts are fine but sustained excess isn't.

Distributed Rate Limiting

In a multi-server deployment, the limit must be global, not per-server — otherwise a client gets limit × server_count. The counter/bucket state must be shared, typically in Redis with atomic operations:

Naive: each of 5 servers enforces "100/min" locally → client gets 500/min. 💥
Correct: shared Redis token bucket (atomic Lua script) → global 100/min. ✅

Client-Side: Respecting Downstream Limits

When calling a rate-limited service, the pattern protects you from being throttled:

- Pace outbound calls with a local token bucket sized to the API's limit.
- Honor 429 responses: read Retry-After and back off (see retry-pattern.md).
- Queue and drain requests at the allowed rate rather than bursting and
  getting rejected.
This keeps you a good citizen and avoids bans/throttling.

Rate Limiter vs Related Patterns

Rate limiter:  cap the RATE of operations (per client/key/global).
Throttling:    slow down / delay excess (a rate-limiter response mode).
Load shedding: drop LOW-PRIORITY work when the SYSTEM is overloaded
               (system-driven, see graceful-degradation.md).
Bulkhead:      isolate RESOURCES so one workload can't exhaust others.

These compose: rate limiting caps per-client volume, bulkheads isolate resources, and load shedding drops low-priority work under system-wide overload.

Seeing It in Action

Scenario: Rate limiting in both directions for an API integration service.

# SERVER-SIDE — protect our API from clients (distributed via Redis).
#   Per-API-key token bucket enforced at the gateway, GLOBAL across servers.
def check_inbound(api_key, plan_rate):
    key = f"ratelimit:{api_key}"
    if redis_token_bucket_allow(key, capacity=plan_rate, refill=plan_rate/60):
        return True                        # within quota → proceed
    raise RateLimited(retry_after=8)       # 429 + Retry-After

# CLIENT-SIDE — respect a third-party API's limit (say, 10 req/sec).
#   Pace our OUTBOUND calls so we're never throttled or banned.
outbound = TokenBucket(capacity=10, refill_rate=10)   # match their limit

def call_third_party(request):
    while not outbound.allow():
        time.sleep(0.05)                   # wait for a token (pace ourselves)
    resp = third_party.send(request)
    if resp.status == 429:                 # they throttled us anyway
        time.sleep(resp.retry_after)       # honor Retry-After, back off
        return call_third_party(request)   # retry after backoff
    return resp

# Why both directions matter:
#  Inbound:  a buggy client looping infinitely hits its per-key limit → gets
#            429s → can't overwhelm our service or degrade other customers.
#            Distributed Redis state means it can't bypass the limit by
#            spreading calls across our servers.
#  Outbound: we never exceed the third party's 10/sec → we're never banned
#            or throttled, and our integration stays reliable. If they
#            throttle us anyway, we honor Retry-After instead of hammering.

Why rate limiting is a fundamental protective pattern in both directions: rate limiting is often thought of only as a server-side defense — capping how much any one client can do so that abuse, bugs, or traffic spikes can't overwhelm your service or let one noisy client degrade everyone else. That inbound use is essential (and, done right, must be distributed — enforced against shared state so a client can't multiply its allowance by spreading requests across your servers, and transparent via 429/Retry-After so well-behaved clients self-regulate). But the same pattern is just as important pointed outward: whenever your system calls a rate-limited dependency — a third-party API, a partner service, even an internal service with quotas — you need to pace your own outbound calls to stay within the allowed rate, or you'll get throttled, rejected, or banned, making your integration unreliable. The token bucket algorithm serves both directions identically because the underlying need is the same: allow controlled bursts up to a capacity while enforcing a sustained average rate. And rate limiting doesn't stand alone — it composes with the other resilience patterns into a coherent defense: rate limiters cap per-client volume, bulkheads isolate resources so one workload can't starve others, circuit breakers stop calling broken dependencies, retries with backoff absorb transient blips (and honor rate-limit responses), and load shedding drops low-priority work under system-wide overload. Each addresses a different way a system gets overwhelmed, and together they let a service protect itself from clients, protect its dependencies from itself, and stay responsive under stress.

Interview Questions

1. Q: What are the two directions of rate limiting, and why does each matter? Hint: Server-side (inbound) protects your service from clients — capping per-client volume so abuse, bugs, or spikes can't overwhelm you or let one client degrade others. Client-side (outbound) paces your calls to a rate-limited downstream (third-party/partner API) so you don't get throttled or banned. The same algorithms serve both; the direction changes what you're protecting — your service vs your integration's reliability.

2. Q: Why is the token bucket the most common rate-limiting algorithm? Hint: It allows controlled bursts (up to the bucket capacity) while enforcing a sustained average rate (the refill rate), matching real usage where occasional bursts are fine but sustained excess isn't. It's memory-efficient (just a token count and timestamp) and simple to implement. Leaky bucket smooths to a constant rate (no bursts), and fixed-window has boundary-burst problems.

3. Q: How do you implement rate limiting correctly across multiple servers? Hint: The limit must be global, using shared state (typically Redis with atomic operations / a Lua token-bucket script), not enforced per-server locally. A naive per-server limit lets a client multiply its allowance by the number of servers (5 servers × 100 = 500). Shared atomic counters/buckets ensure the client sees one global limit regardless of which server handles each request.

4. Q: When calling a rate-limited third-party API, how does the rate limiter pattern help you? Hint: You pace your outbound calls with a local token bucket sized to the API's limit, so you stay within it and aren't throttled or banned. You also honor 429/Retry-After responses by backing off rather than hammering, and queue/drain requests at the allowed rate instead of bursting and getting rejected. This keeps your integration reliable and makes you a good API citizen.

5. Q: How does rate limiting compose with bulkheads, circuit breakers, and load shedding? Hint: They address different overload modes and combine into a resilience toolkit: rate limiters cap per-client/global request volume; bulkheads isolate resources so one workload can't exhaust others; circuit breakers stop calling failing dependencies; retries (with backoff/jitter) absorb transient blips and honor rate-limit responses; load shedding drops low-priority work under system-wide overload. Together they let a service protect itself, its dependencies, and its responsiveness.

References

• Cloudflare: What is rate limiting? — algorithms and concepts
• Stripe: Scaling your API with rate limiters — token bucket with Redis in production
• Topic 11 — Rate Limiting & Throttling — the API-design view

Dive Deeper

• System Design: Designing a rate limiter (ByteByteGo) — algorithm comparison in depth
• Resilience4j RateLimiter — a client-side rate limiter implementation
• Redis rate limiting patterns — distributed counters and token buckets