Circuit Breaker

7 min read

In a Nutshell

A circuit breaker protects your system from a failing dependency, exactly like the electrical breaker in your house protects your wiring from a surge. When a downstream service (a database, a payment API, another microservice) starts failing or timing out, the circuit breaker "trips" and immediately rejects further calls instead of letting them pile up. This stops a slow or dead dependency from exhausting your threads, connections, and memory — the mechanism that turns one service's failure into a cascading, system-wide outage. After a cooldown, it cautiously tests whether the dependency has recovered before resuming normal traffic.

2D minimalistic diagram showing a service calling a downstream dependency through a circuit breaker component; the dependency is failing with red X marks, and the breaker is shown "open" (like an open electrical switch) blocking the calls and returning a fast fallback response instead of hanging

How It Actually Works

The Problem It Solves: Cascading Failure

Without a circuit breaker, a slow dependency drags everything down:

Payment API becomes slow (10s per call instead of 50ms).
   │
Your service keeps calling it; each call holds a thread for 10s.
   │
Thread pool exhausts → your service can't handle ANY request,
   even ones that don't need payments.
   │
Upstream callers time out waiting for you → THEY exhaust too.
   │
💥 One slow dependency cascades into a full outage.

The circuit breaker breaks this chain: once it detects the dependency is unhealthy, it fails fast (returns immediately) instead of tying up resources waiting.

The Three States

        failures exceed threshold
   ┌──────────────────────────────────┐
   │                                   ▼
┌──────┐  timeout elapsed        ┌──────────┐
│CLOSED│◀───── success ──────────│ HALF-OPEN│
└──────┘                         └──────────┘
   ▲   │                              │  ▲
   │   │ trip (too many failures)     │  │ test call fails
   │   ▼                              ▼  │
   │  ┌──────┐   after cooldown       │
   └──│ OPEN │───────────────────────┘
      └──────┘
      reject all calls immediately (fail fast)

State	Behavior	Transition
Closed	Calls pass through normally; failures are counted	Trips to Open when failure rate/count exceeds threshold
Open	Calls fail immediately (no downstream request)	After a cooldown timer, moves to Half-Open
Half-Open	Allows a limited number of trial calls	Success → Closed; failure → back to Open

Key Tuning Parameters

Parameter	Controls	If Too Low	If Too High
Failure threshold	When to trip	Trips on transient blips	Slow to protect; damage done
Rolling window	Over what period failures count	Noisy	Sluggish reaction
Open/cooldown duration	How long to stay tripped	Hammers a recovering service	Slow recovery
Half-open trial count	How many probes before re-closing	Premature re-close	Slow to recover

Circuit Breaker vs Retry vs Timeout

These three resilience patterns work together, not in competition:

Pattern	Job	Danger If Used Alone
Timeout	Bound how long you wait	Without it, calls hang forever
Retry	Handle transient blips	Retries amplify load on a struggling service
Circuit Breaker	Stop calling a broken service entirely	—

The correct combination: timeout every call, retry a couple of times with backoff for transient errors, and wrap the whole thing in a circuit breaker so that when errors are persistent, you stop retrying and fail fast. Retry without a circuit breaker is dangerous — it piles more load on an already-failing dependency.

The Fallback

When the breaker is open, you don't have to return an error — you can degrade gracefully (see Graceful Degradation):

Return cached/stale data
Return a sensible default (empty recommendations, "try again later")
Queue the request for later processing
Route to a backup provider

2D minimalistic state-machine diagram of the circuit breaker's three states — Closed, Open, Half-Open — drawn as three circles with labeled arrows: "failures exceed threshold" (Closed→Open), "cooldown elapsed" (Open→Half-Open), "trial succeeds" (Half-Open→Closed), and "trial fails" (Half-Open→Open), styled like an electrical switch diagram

Seeing It in Action

Scenario: Protecting a checkout service from a flaky recommendations API.

from pybreaker import CircuitBreaker

# Trip after 5 failures; stay open 30s before testing recovery.
rec_breaker = CircuitBreaker(fail_max=5, reset_timeout=30)

@rec_breaker
def _fetch_recommendations(user_id):
    # Every call is also timeout-bounded so a hang counts as a failure.
    return recommendations_api.get(user_id, timeout=0.5)

def get_recommendations(user_id):
    try:
        return _fetch_recommendations(user_id)
    except CircuitBreakerError:
        # Breaker is OPEN — fail fast, don't even attempt the call.
        return cached_popular_items()          # graceful fallback
    except (Timeout, APIError):
        return cached_popular_items()

# Result:
#  - Normal:   real personalized recommendations
#  - API slow: after 5 failures the breaker opens; checkout stays fast
#              and shows popular items instead of hanging
#  - Recovery: after 30s, one trial call decides whether to resume

Why this matters: the recommendations API is non-critical — checkout must never hang or fail because recommendations are down. The circuit breaker guarantees that a failure in a nice-to-have dependency degrades to "show popular items" instead of taking down the entire checkout flow.

Interview Questions

Q: What problem does the circuit breaker pattern solve? Hint: It prevents a failing/slow dependency from cascading into a full outage. Without it, calls to a slow service tie up threads/connections until the caller's resources exhaust, which propagates upstream. The breaker detects persistent failures and "fails fast" — rejecting calls immediately so resources aren't consumed waiting on a broken dependency.
Q: Describe the three states of a circuit breaker and the transitions. Hint: Closed (calls flow, failures counted; trips to Open when threshold exceeded), Open (calls rejected immediately for a cooldown period, then moves to Half-Open), Half-Open (a few trial calls; success → Closed, failure → back to Open). This lets the system recover automatically without hammering a still-broken dependency.
Q: Why is retrying without a circuit breaker dangerous? Hint: Retries add more load to an already-struggling service, accelerating its collapse and prolonging the outage (retry storms). Combine retries (with backoff + jitter) for transient errors with a circuit breaker that stops retrying when failures are persistent — plus timeouts so slow calls count as failures.
Q: How do timeout, retry, and circuit breaker work together? Hint: Timeout bounds each call so it can't hang. Retry (with backoff/jitter) handles transient blips. Circuit breaker detects persistent failure and stops calling entirely, failing fast. Layered: timeout every call → retry a couple times → trip the breaker when errors persist → serve a fallback.
Q: What should happen when the breaker is open — just return an error? Hint: Ideally degrade gracefully rather than erroring: serve cached/stale data, a sensible default, queue for later, or route to a backup provider. The right fallback depends on whether the dependency is critical. For non-critical features (recommendations), a fallback keeps the core flow working; for critical ones, you may have to surface an error but at least fast.

References

Martin Fowler: CircuitBreaker — the canonical explanation
Release It! by Michael Nygard — origin of the pattern (stability patterns)
Resilience4j documentation — a production circuit-breaker library

Dive Deeper

Netflix Hystrix (archived) & its lessons — circuit breaking at massive scale
Google SRE Book — Addressing Cascading Failures — the failure modes breakers prevent
AWS Builders' Library: Timeouts, retries, and backoff with jitter — the companion patterns