Availability & Reliability
Staying Up When Things Go Wrong
Scalability is about handling growth; availability and reliability are about handling failure — because in any system of meaningful size, failure is not an exception, it's a constant. Disks die, networks partition, deployments go wrong, and entire data centers occasionally go dark. Reliability is the property of consistently doing the right thing; availability is the property of being there to do it. Together they answer the question every real system must: when — not if — something breaks, what happens to the user?
The discipline here is a shift in mindset from "prevent failure" to "expect failure and contain it." You assume every component will fail, then design so that no single failure — and ideally no small cluster of failures — can bring down the whole system or lose committed data. This is achieved through a layered toolkit: redundancy and failover for component failures, disaster recovery for catastrophic ones, durable storage and backups to protect data, careful consistency guarantees so replicated data stays correct, and defensive patterns like circuit breakers and graceful degradation so one failing part doesn't drag down the rest.
When This Comes Up
- System design interviews: Once you've sketched a scalable design, the follow-ups are relentlessly about failure: "What happens when this database goes down?" "How do you avoid losing data?" "What if this whole region fails?" Strong candidates proactively identify single points of failure, state their availability target, and reach for the right pattern (failover, multi-AZ, circuit breaker) with its RTO/RPO implications.
- Real architecture: SLAs and error budgets turn availability into a contractual, measurable number. Every design decision — how many replicas, sync vs async, how backups are taken and tested — is ultimately about hitting a reliability target at acceptable cost.
- Production incidents: This is the topic you live at 3am. Cascading failures, failed failovers, corrupt backups, split-brain, and overload collapse are all here — and the patterns in this topic are precisely what prevent a small fault from becoming a company-wide outage.
How the Sub-Topics Connect
The sub-topics move from the goal (high availability) → the mechanism that sustains it (fault tolerance and failover) → surviving catastrophe (disaster recovery) → protecting the data itself (backups and durability) → keeping replicated data correct (consistency) → and finally the defensive patterns that contain failure at the application layer (circuit breaker, graceful degradation):
1. High Availability
The goal: keep serving even when components fail, measured in "nines" (99.9% is ~8.7 hours of downtime a year; 99.999% is ~5 minutes). You get there by eliminating single points of failure — every critical component gets redundancy, and traffic automatically routes away from anything unhealthy. The availability math matters: components in series multiply (making the whole less available), while redundant components in parallel dramatically increase it. But redundancy alone isn't enough — you also need automatic detection and recovery, because if failover needs a human, your availability is capped by human response time.
2. Fault Tolerance & Failover
The mechanism behind HA: fault tolerance is the ability to keep operating correctly when parts fail, and failover is the automatic switch to a healthy backup. Precise vocabulary helps — a fault (component defect) becomes an error (bad state) becomes a failure (wrong service), and the job is to stop that chain. The failover sequence (detect → confirm → fence → promote → reroute → recover) has two hard steps: confirming dead-vs-slow to avoid needless failover, and fencing the old node to prevent split-brain. Watch for failover storms, flapping, and the classic sin of untested failover.
3. Disaster Recovery
Where fault tolerance handles routine component failure, DR handles catastrophe — a whole region gone, a data-center fire, ransomware. Two numbers drive every decision: RTO (how fast you must recover) and RPO (how much data you can afford to lose). The four strategies trade cost against those numbers: backup-and-restore (cheap, slow) → pilot light → warm standby → multi-site active-active (near-zero RTO/RPO, expensive). The rules that make DR real: the 3-2-1 backup rule, geographic isolation, automated runbooks, and — above all — regularly testing your recovery, because an untested plan is just a hope.
4. Backup & Data Durability
Durability guarantees committed data survives hardware failure (via replication, erasure coding, write-ahead logs, and fsync); backups are point-in-time copies that let you recover from logical loss — accidental deletes, bad migrations, ransomware. The single most important lesson: replication is not a backup, because it faithfully replicates your mistakes to every copy. Point-in-time recovery (base backup + continuous change log) is the gold standard, letting you rewind to the second before disaster. And the most-skipped, most-critical step is verifying backups by actually restoring them.
5. Data Consistency
Reliability isn't just staying up — it's serving correct data. Consistency is the guarantee that everyone reading replicated data sees an agreed-upon view, and it's a spectrum from eventual (cheap, fast, available) to strong/linearizable (expensive, coordinated, correct). The core trade-off is CAP's: during a partition, choose consistency or availability. Strong consistency uses consensus or quorums (R + W > N); eventual consistency needs conflict resolution (LWW, version vectors, CRDTs). The pragmatic reality is that consistency is a per-feature decision — strong for payments, eventual for like counts.
6. Circuit Breaker
A defensive pattern that stops a failing dependency from cascading into a system-wide outage — just like an electrical breaker. When a downstream service starts failing or timing out, the breaker "trips" and rejects calls immediately (fails fast) instead of letting them exhaust threads and connections. It cycles through three states — Closed, Open, Half-Open — automatically testing for recovery after a cooldown. It composes with timeouts and retries (retry without a breaker is dangerous — it piles load on a struggling service) and pairs naturally with a graceful fallback.
7. Graceful Degradation
The strategy for what the user experiences when something is unavailable: shed non-essential features while keeping the core working, so failures become partial and invisible rather than total. A video site should still play videos when recommendations are down; an e-commerce site should still sell when reviews are down. Techniques include serving stale data, feature kill switches, default responses, reduced fidelity, and load shedding (deliberately rejecting low-priority requests under overload so the important ones succeed). The essential discipline: log every degradation, so running degraded doesn't become an invisible outage.
Sub-Topics
| # | Sub-Topic | What You'll Learn |
|---|---|---|
| 1 | High Availability | Designing for uptime with redundancy and the availability math of nines |
| 2 | Fault Tolerance & Failover | Surviving component failure and switching to backups safely |
| 3 | Disaster Recovery | RTO/RPO and the four strategies for surviving catastrophe |
| 4 | Backup & Data Durability | Protecting data from hardware and logical loss — and why replication isn't a backup |
| 5 | Data Consistency | Keeping replicated data correct across the consistency spectrum |
| 6 | Circuit Breaker | Stopping a failing dependency from cascading into an outage |
| 7 | Graceful Degradation | Shedding non-essential features to keep the core experience alive |