Fault Tolerance & Failover

7 min read

In a Nutshell

Fault tolerance is a system's ability to keep operating correctly even when parts of it fail. Failover is the specific mechanism that makes it happen: detecting a failure and automatically switching to a healthy backup. The distinction from high availability is subtle — HA is the goal (stay up), fault tolerance is the property (survive faults), and failover is the technique (switch to a standby). A fault-tolerant system anticipates failures — dead servers, corrupted disks, network partitions — and has a pre-planned, automatic response so that a fault becomes a non-event rather than an outage.

2D minimalistic diagram showing a primary node handling traffic with a healthy heartbeat line to a monitor; the primary then fails with a red X, the heartbeat flatlines, and an arrow labeled "failover" swings traffic over to a standby node that becomes the new primary

How It Actually Works

Faults, Errors, and Failures

Precise vocabulary matters here:

Term	Meaning	Example
Fault	A component deviates from spec	A disk sector goes bad
Error	The fault manifests in system state	A read returns corrupted data
Failure	The system stops delivering correct service	The API returns wrong results to users

Fault tolerance is about stopping faults from cascading into failures. The goal is graceful handling, not the impossible dream of a fault-free system.

Types of Faults

Fault Type	Description	Handling Strategy
Crash (fail-stop)	Node halts cleanly	Failover to replica
Omission	Messages/responses lost	Retries, timeouts
Timing	Response too slow	Timeouts, hedged requests
Byzantine	Component behaves arbitrarily/maliciously	BFT consensus (rare, expensive)

Most systems assume fail-stop faults (nodes crash cleanly) because Byzantine fault tolerance is far more expensive and only needed for adversarial environments like blockchains.

The Failover Sequence

1. DETECT     health checks / heartbeats miss N intervals
                 │
2. CONFIRM     avoid false positives — require multiple missed beats
                 │  (a slow node ≠ a dead node)
3. FENCE       ensure the old primary can't keep writing (STONITH / epoch token)
                 │
4. PROMOTE     select and activate the standby as new primary
                 │
5. REROUTE     update DNS / LB / service registry to point at new primary
                 │
6. RECOVER     rebuild the failed node as a fresh standby

The two hardest steps are CONFIRM (distinguishing dead from slow to avoid needless failover) and FENCE (preventing the old node from causing split-brain — see Consensus & Leader Election).

Redundancy Patterns for Fault Tolerance

Pattern	How It Works	Trade-off
Hot standby	Backup fully running, synced, ready instantly	Fast failover, costs full duplicate
Warm standby	Backup running but not fully synced	Cheaper, slightly slower failover
Cold standby	Backup provisioned only on failure	Cheapest, slowest recovery
N-version / redundancy	Multiple independent implementations vote	Tolerates correlated bugs, very costly

Failover Anti-Patterns

Failover storms — a wave of failovers overwhelms the surviving nodes, causing them to fail too (cascading failure). Mitigate with load shedding and circuit breakers.
Flapping — repeatedly failing over and back due to a marginal component. Mitigate with hysteresis (require sustained health/failure before switching).
Untested failover — the backup was never exercised and doesn't actually work. Mitigate with regular game days / chaos testing.
Failover that loses data — async-replicated writes vanish on promotion. Mitigate with semi-sync replication and accepting a defined RPO.

2D minimalistic diagram comparing hot, warm, and cold standby: three rows each showing a primary and a standby, where "hot" has the standby fully lit and synced with a fast failover arrow, "warm" has it partially lit with a medium arrow, and "cold" has it powered off with a slow "provision + start" arrow, annotated with recovery time and cost

Seeing It in Action

Scenario: Database failover with a coordinator (e.g., Patroni/orchestrator for Postgres/MySQL).

Normal operation:
  App ──writes──▶ [Primary]──replicates──▶ [Replica-1], [Replica-2]
                     │
                  heartbeat every 1s ──▶ [Coordinator watches all nodes]

Primary crashes:
  1. Coordinator misses 3 heartbeats (3s) → suspects failure
  2. Confirms replica also can't reach primary → confirmed down
  3. Fences old primary: revokes its lease / blocks its writes (epoch bump)
  4. Picks Replica-1 (least replication lag) → promotes to Primary
  5. Updates the service registry; app reconnects to new Primary
  6. Replica-2 re-points to the new Primary
  7. Old primary, when it recovers, rejoins as a Replica (never as Primary)

Total downtime: a few seconds of write unavailability.
Reads from replicas may continue throughout.

Why the fence matters: without step 3, if the "crashed" primary was merely partitioned and comes back thinking it's still in charge, you'd have two primaries accepting conflicting writes — split-brain and data corruption. The epoch/fencing token guarantees the old primary's writes are rejected.

Interview Questions

Q: Distinguish fault, error, and failure. Why does the distinction matter? Hint: A fault is a component defect (bad disk sector), an error is that fault reflected in state (corrupted read), a failure is the system delivering incorrect service to users. Fault tolerance aims to stop faults/errors from becoming user-visible failures — you can't prevent all faults, but you can contain them.
Q: What's the difference between fail-stop and Byzantine faults, and why do most systems assume fail-stop? Hint: Fail-stop: a node halts cleanly and detectably. Byzantine: a node behaves arbitrarily or maliciously (wrong values, lies). Byzantine fault tolerance requires expensive protocols (BFT consensus, ≥3f+1 nodes) and is only needed in adversarial settings (blockchains). Most internal systems assume fail-stop because it's far cheaper and usually sufficient.
Q: Walk through a safe database failover. What are the two hardest steps? Hint: Detect (missed heartbeats) → confirm (avoid acting on a slow-but-alive node) → fence the old primary → promote the most up-to-date replica → reroute clients → rebuild the old node as a replica. Hardest: confirming (dead vs slow, to avoid needless failover) and fencing (prevent split-brain).
Q: What is a failover storm / cascading failure, and how do you prevent it? Hint: When one node fails, its load shifts to survivors, which then overload and fail too, cascading. Prevent with capacity headroom, load shedding, circuit breakers, backpressure, and rate limiting so surviving nodes protect themselves rather than accepting fatal overload.
Q: Compare hot, warm, and cold standby. Hint: Hot = fully running and synced, near-instant failover, full duplicate cost. Warm = running but not fully caught up, moderate cost and failover time. Cold = provisioned only on failure, cheapest, slowest recovery. Choose based on how much downtime (RTO) and data loss (RPO) you can tolerate versus budget.

References

Designing Data-Intensive Applications by Martin Kleppmann — Chapters 5 & 8: Faults and Failover
Patroni — HA/failover template for PostgreSQL
Google SRE Book — Addressing Cascading Failures — failover storms and prevention

Dive Deeper

GitHub orchestrator — production MySQL failover and its subtleties
Fallacies of Distributed Computing — the assumptions that cause failover bugs
Jepsen: split-brain analyses — what happens when fencing goes wrong