Replication

8 min read

In a Nutshell

Replication copies data to multiple machines for three purposes: durability (data survives machine failures), availability (the system keeps serving if a node goes down), and read scaling (distribute reads across copies). The fundamental trade-off is between consistency (how quickly replicas reflect the latest write) and performance (how fast writes are acknowledged). There are three main replication topologies — single-leader, multi-leader, and leaderless — each with distinct strengths and failure modes. Understanding when to use which is essential for designing systems that are both reliable and performant.

2D minimalistic diagram showing a primary database node labeled "Leader" sending arrows to three replica nodes labeled "Follower 1, 2, 3", with write arrows going to the leader only and read arrows going to all followers

How It Actually Works

Single-Leader Replication (Primary-Replica)

The most common model. One node (the leader/primary) handles all writes. Replicas (followers) receive a copy of every write via a replication log.

Client writes ──▶ Leader (Primary)
                     │
              replication log
              ┌──────┼──────┐
              ▼      ▼      ▼
          Replica  Replica  Replica
          (read)   (read)   (read)
              ▲      ▲      ▲
Client reads ─┘      │      │
Client reads ────────┘      │
Client reads ───────────────┘

Aspect	Description
Writes	Go to the leader only
Reads	Can go to any replica (or the leader)
Replication mode	Synchronous or asynchronous
Failover	If leader dies, promote a replica to leader
Used by	PostgreSQL, MySQL, MongoDB, Redis Sentinel

Sync vs Async Replication

Mode	How It Works	Trade-off
Synchronous	Write not acknowledged until ≥1 replica confirms	Strong consistency, but higher write latency and lower availability (if sync replica is down, writes block)
Asynchronous	Write acknowledged as soon as leader writes locally	Fast writes, but replicas may lag — reads from replicas can be stale
Semi-synchronous	At least 1 replica is sync, others are async	Balance: one guaranteed up-to-date replica, others eventually consistent

Replication Lag

The delay between a write on the leader and when it appears on a replica. Causes:

Network latency between leader and replica
Replica is under heavy read load (can't keep up with replication)
Large transactions (long-running writes block replication)

Impact: A user writes data, then reads from a replica that hasn't received the write yet → stale read. See Consistency Models for how to handle this.

Multi-Leader Replication

Multiple nodes accept writes. Each leader replicates to all others.

Leader A ◀──────────▶ Leader B
(Region: US)          (Region: EU)
   │                     │
   ▼                     ▼
Replicas              Replicas

Pros	Cons
Lower write latency (write to local leader)	Write conflicts — two leaders modify the same data simultaneously
Higher availability (each region is independent)	Conflict resolution is complex (last-write-wins? merge? manual?)
Multi-region support	Harder to reason about consistency

Used by: CouchDB, multi-datacenter MySQL/PostgreSQL setups, collaborative editing (Google Docs uses a variant).

The hard problem — write conflicts:

Leader A: UPDATE users SET name = 'Alice Smith' WHERE id = 123;  (at T1)
Leader B: UPDATE users SET name = 'Alice Johnson' WHERE id = 123; (at T1)

Both succeed locally. When they replicate to each other:
  - Last-Write-Wins (LWW): Compare timestamps, keep the latest → data loss
  - Custom merge: Application-specific logic (e.g., concatenate changes)
  - Flag for manual resolution: Alert the user to resolve the conflict

Leaderless Replication (Dynamo-Style)

No designated leader. Writes and reads go to multiple nodes simultaneously. Consistency is achieved through quorum reads/writes.

Client write ──▶ Node 1 ✓ (success)
               ──▶ Node 2 ✓ (success)
               ──▶ Node 3 ✗ (down — but quorum met: 2 of 3)

Client read  ──▶ Node 1: value = v2
             ──▶ Node 3: value = v1 (stale — was down during write)
             → Return v2 (latest) and trigger read repair on Node 3

Quorum formula: W + R > N guarantees a read will see the latest write.

N = number of replicas (e.g., 3)
W = number of write confirmations required (e.g., 2)
R = number of read confirmations required (e.g., 2)
W + R > N → at least one node has the latest value

Pros	Cons
No single point of failure (no leader)	No global ordering of writes → conflicts possible
High availability (tolerates node failures)	Read repair and anti-entropy add complexity
Tunable consistency (adjust W and R)	Sloppy quorums can violate consistency guarantees

Used by: Cassandra, DynamoDB, Riak, Voldemort.

2D minimalistic comparison of three replication models side by side: single-leader (one write node, multiple read nodes), multi-leader (multiple write nodes replicating to each other), leaderless (all nodes accept reads and writes with quorum)

Failover

When the leader dies in a single-leader setup:

Detection — Followers notice the leader hasn't sent a heartbeat (timeout: 10–30 seconds)
Election — Followers agree on a new leader (via Raft, Paxos, or manual promotion)
Reconfiguration — Clients and other replicas redirect writes to the new leader
Data reconciliation — If the old leader had unreplicated writes, they're lost (async) or the old leader is stopped (sync)

Danger: Split-brain — If the old leader comes back and both it and the new leader accept writes, you have two divergent copies. Fencing (STONITH — "Shoot The Other Node In The Head") prevents this by ensuring the old leader is terminated.

Seeing It in Action

Scenario: Designing replication for a global e-commerce platform

Architecture:
┌────────────────────────────────────────────────────────┐
│                   US-East Region                       │
│  ┌──────────────┐                                     │
│  │  Primary DB   │─── sync ──▶ Replica (same AZ)     │
│  │  (Leader)     │─── async ─▶ Replica (diff AZ)     │
│  └──────┬────────┘                                     │
│         │ async                                        │
└─────────┼──────────────────────────────────────────────┘
          │
    cross-region async replication
          │
┌─────────▼──────────────────────────────────────────────┐
│                   EU-West Region                       │
│  ┌──────────────┐                                     │
│  │  Read Replica │ (serves EU reads with low latency)  │
│  └──────────────┘                                     │
│  On primary failure: promote to leader (manual/auto)   │
└────────────────────────────────────────────────────────┘

Design decisions:

Semi-sync in same region: one sync replica (guaranteed up-to-date for failover), one async (performance)
Async cross-region: ~100ms latency between US and EU makes sync impractical
EU reads from local replica: low latency for EU users (eventual consistency acceptable for product catalog)
Writes always go to US primary: strong consistency for inventory and orders
Failover plan: if US primary dies, promote the sync replica in the same AZ. If the entire US region fails, promote the EU replica (accepting some data loss from async lag)

Interview Questions

Q: What's the difference between synchronous and asynchronous replication? When would you use each? Hint: Sync: write waits for replica confirmation — no data loss on leader failure, but higher latency and availability risk if the replica is slow. Async: write acknowledged immediately — fast, but replica may lag, causing stale reads and potential data loss on failover. Use sync for critical data (financial); async for read scaling with acceptable staleness.
Q: How does a quorum work in leaderless replication? What values of W and R would you choose? Hint: W + R > N ensures overlap (at least one node in the read set has the latest write). With N=3: W=2, R=2 is the standard (tolerates 1 failure). W=1, R=3 prioritizes write speed. W=3, R=1 prioritizes read speed. W=1, R=1 gives no consistency guarantee — pure eventual.
Q: What is split-brain, and how do you prevent it? Hint: Two nodes both believe they're the leader and accept writes — creating divergent data. Prevention: fencing tokens (new leader invalidates old leader's token), STONITH (terminate the old leader), or consensus protocols (Raft/Paxos ensure only one leader). Detection: epoch numbers that increment on each leader election.
Q: Your replica has 30 seconds of replication lag. A user writes data and immediately reads it from the replica. What happens, and how do you fix it? Hint: The user sees stale data (their write hasn't replicated yet). Fixes: 1) Read-your-writes consistency: route the user's reads to the leader after a write. 2) Sticky sessions: always route a user to the same replica. 3) Causal tokens: track the write's position in the replication log and wait for the replica to reach it.
Q: When would you choose multi-leader replication over single-leader? Hint: Multi-region deployment where write latency to a single leader in another region is unacceptable. Example: a collaborative editing app where users in Tokyo and London both need fast writes. Trade-off: write conflicts must be resolved (LWW, merge, or manual). If you can tolerate all writes going to one region, single-leader is simpler and avoids conflicts.

References

Designing Data-Intensive Applications by Martin Kleppmann — Chapter 5: Replication
PostgreSQL Streaming Replication — official replication docs
MySQL Replication — MySQL replication setup and monitoring

Dive Deeper

Raft Consensus Algorithm — interactive visualization of leader election and log replication
Amazon Dynamo Paper — the foundational paper on leaderless replication
GitHub — MySQL High Availability — real-world replication and failover at scale