High Availability

6 min read

In a Nutshell

High availability (HA) is designing a system so it keeps serving requests even when individual components fail. It's measured in "nines": 99.9% ("three nines") allows ~8.7 hours of downtime a year, while 99.999% ("five nines") allows only ~5 minutes. You achieve HA by eliminating single points of failure — every critical component gets redundancy, and traffic automatically routes away from anything unhealthy. The core idea is simple but demanding: assume every piece will fail, and make sure no single failure can take down the whole system.

2D minimalistic diagram showing a system with redundant components at every layer: two load balancers, three app servers, and a primary-plus-replica database, all with a red X striking through one component at each layer while arrows show traffic rerouting around the failures to keep the system serving

How It Actually Works

The Nines and What They Cost

Availability	Downtime/Year	Downtime/Month	Typical Use
99% (two nines)	3.65 days	7.2 hours	Internal tools
99.9% (three nines)	8.77 hours	43.8 min	Standard SaaS
99.99% (four nines)	52.6 min	4.4 min	E-commerce, business-critical
99.999% (five nines)	5.26 min	26 sec	Telecom, payments, core infra

Each additional nine is dramatically more expensive — it demands more redundancy, faster failover, multi-region deployment, and rigorous operational discipline. Don't promise five nines when your business needs three.

Availability Math: Series vs Parallel

Components in series (all must work) multiply their availabilities, making the whole less available than any part:

Series (dependency chain):  A_total = A1 × A2 × A3
  LB(99.9%) → App(99.9%) → DB(99.9%) = 0.999³ ≈ 99.7%   (worse!)

Redundant components in parallel (any one suffices) make the whole more available:

Parallel (redundancy):  A_total = 1 − (1−A1)(1−A2)
  Two app servers at 99% each = 1 − (0.01 × 0.01) = 99.99%   (better!)

This is the mathematical case for redundancy: adding a parallel copy of a 99% component yields 99.99%.

Eliminating Single Points of Failure (SPOFs)

A SPOF is any component whose failure takes down the system. The HA method is to find every SPOF and add redundancy:

Layer	SPOF	HA Fix
DNS	Single provider	Multiple DNS providers
Load balancer	One LB	Redundant LBs + virtual IP (VRRP)
App tier	One server	N stateless servers behind LB
Database	Single primary	Primary + replicas with auto-failover
Zone	Single data center	Multi-AZ deployment
Region	Single region	Multi-region (active-active or active-passive)

Redundancy Models

Model	Standby Behavior	Failover Time	Cost
Active-Passive	Standby idle until primary fails	Seconds–minutes	Lower (idle capacity)
Active-Active	All nodes serve traffic	Near-instant	Higher (but no wasted capacity)
N+1 / N+M	M spare units for N active	Fast	Moderate

Redundancy Isn't Enough — You Need Automatic Detection & Recovery

Redundancy only helps if the system notices a failure and reacts without a human:

Detect  →  Isolate  →  Reroute  →  Recover
  │           │           │           │
health    remove bad   send traffic  rebuild/replace
checks    node from    to healthy    the failed
+heartbeat  pool       redundant     component
                       node

If failover requires paging an engineer, your real availability is bounded by human response time — not by your redundancy.

2D minimalistic diagram illustrating series vs parallel availability: top row shows three boxes connected in a line (series) with a formula "0.999 × 0.999 × 0.999 = 99.7%"; bottom row shows two stacked boxes in parallel with a formula "1 − (0.01 × 0.01) = 99.99%", with the parallel result highlighted in green as more available

Seeing It in Action

Scenario: Making a typical web service highly available.

Before (SPOFs everywhere):
  Client → 1 LB → 1 app server → 1 database
  Any single failure = full outage.  Effective ~99% at best.

After (HA architecture):
  Client
    │
    ├─ DNS (2 providers, health-checked)
    │
  [ LB-A ]═══[ LB-B ]        active-active, shared virtual IP
    │            │
  ┌─┴────┬───────┴─┐
  App-1  App-2  App-3        stateless, behind LB, autoscaled
  └──┬───┴───┬────┘
     │       │
  [Primary DB]──replicates──[Replica DB]   auto-failover on primary loss
     (AZ-1)                    (AZ-2)        spread across availability zones

  Result: no single component failure causes an outage.
  Combined with multi-AZ, tolerates a full data-center loss.

The availability budget in practice: if you target 99.99% (52 min/year), you must account for planned work too — deployments, patching, migrations. This is why HA architectures use rolling deployments and can drain/replace nodes without downtime; otherwise routine maintenance eats your entire error budget.

Interview Questions

Q: What does "five nines" mean, and why isn't it always the right target? Hint: 99.999% availability ≈ 5.26 minutes of downtime per year. Each added nine costs dramatically more (more redundancy, faster failover, multi-region, operational rigor). Over-provisioning availability wastes money; match the target to business need — payments/telecom need five nines, an internal dashboard is fine at three.
Q: How does adding a redundant component improve availability mathematically? Hint: Series components multiply availabilities (chain is less available than any link). Parallel/redundant components combine as 1−∏(1−Aᵢ), so two 99% components in parallel give 99.99%. Redundancy converts "any one must survive" into much higher combined availability.
Q: What's the difference between active-active and active-passive redundancy? Hint: Active-active: all nodes serve traffic, near-instant failover, no wasted capacity, but requires the system to handle concurrent multi-node operation (and conflicts for stateful tiers). Active-passive: standby is idle until the primary fails, cheaper in operational complexity but wastes capacity and has slower failover.
Q: Why is redundancy alone insufficient for high availability? Hint: You also need automatic failure detection (health checks, heartbeats) and recovery (isolate the bad node, reroute traffic, rebuild). If failover depends on a human noticing and reacting, availability is capped by human response time regardless of how much redundant hardware you have.
Q: Your architecture is fully redundant within one data center. Are you highly available? What's still a SPOF? Hint: No — the data center (availability zone) itself is a SPOF: power, cooling, network, or a natural disaster can take the whole facility down. Deploy across multiple AZs, and for regional resilience, multiple regions. Also check for hidden SPOFs: shared DNS, a single config service, or a single deployment pipeline.

References

Google SRE Book — Availability — the nines table and error budgets
AWS Well-Architected: Reliability Pillar — HA design patterns
Designing Data-Intensive Applications by Martin Kleppmann — Chapter 1: Reliability

Dive Deeper

Calculating availability of multi-tier architectures — AWS deep dive on availability math
Netflix: Chaos Engineering — proving availability by deliberately breaking things
Site Reliability Engineering by Google — error budgets and the economics of availability