Replication & Read Replicas

In a Nutshell

Replication keeps copies of your data on multiple machines. It buys you three things at once: read scalability (spread read queries across replicas), high availability (promote a replica if the primary dies), and durability (data survives a single machine's failure). The most common pattern is single-leader replication with read replicas: all writes go to one primary, which streams its changes to one or more read-only followers. The catch is replication lag — followers are always a little behind, so a read from a replica may return slightly stale data. Managing that staleness is the central design challenge.

How It Actually Works

The Read-Scaling Insight

Most applications are read-heavy — often 90%+ of queries are reads. A single primary can handle a lot of writes but chokes when reads pile on top. Read replicas break that coupling:

Before (single node handles everything):
  Writes + Reads ─────▶ [ Primary ]   ← saturated by read load

After (reads offloaded):
  Writes ───────────▶ [ Primary ] ──replication──▶ [ Replica 1 ]
  Reads  ──────────────────────────────────────▶  [ Replica 2 ]
                                                   [ Replica 3 ]
  Add replicas to scale reads almost linearly.

Adding replicas scales reads nearly linearly — until the write volume alone saturates the primary (at which point you need sharding, because every replica must also apply every write).

Synchronous vs Asynchronous Replication

	Synchronous	Asynchronous
Commit waits for	Replica to acknowledge	Nothing — primary commits immediately
Data loss on primary failure	None (replica has it)	Possible (unreplicated writes lost)
Write latency	Higher (round trip to replica)	Lower
Availability	Blocks if replica is down	Primary keeps serving

Pure synchronous is rarely used (one slow replica stalls all writes). Pure async risks data loss on failover. The pragmatic middle ground is semi-synchronous: wait for at least one replica to acknowledge, let the rest catch up asynchronously.

Replication Topologies

Topology	Writes	Notes
Single-leader	One primary	Simplest, most common; no write conflicts
Multi-leader	Several primaries	Writes accepted in multiple regions; must resolve conflicts
Leaderless (quorum)	Any node	Dynamo-style; R + W > N for consistency

For scalability specifically, single-leader with read replicas covers the vast majority of cases. Multi-leader shows up for multi-region write locality; leaderless (Cassandra, DynamoDB) for extreme availability.

Replication Lag and the Read-Your-Writes Problem

Because followers apply changes after the primary, a user who writes then immediately reads from a replica may not see their own change:

t0: User updates profile photo   → Primary
t1: Replication lag = 200ms
t1: User reloads page → reads from Replica → sees OLD photo  😱

Mitigations (in increasing strength):

Technique	How	Cost
Read-your-writes routing	Route a user's reads to the primary for a short window after they write	Slight primary load
Monotonic reads	Pin a user's session to one replica so time doesn't "go backward"	Session affinity
Read from primary for critical paths	Financial/consistency-sensitive reads bypass replicas	Less read offload
Track write position (LSN/GTID)	Read waits until the chosen replica has caught up to the write's log position	Complexity

Failover: Promoting a Replica

When the primary dies, a replica is promoted to take its place. This must be automated but careful:

Failover sequence:
  1. Detect primary is down (health checks, missed heartbeats)
  2. Choose the most up-to-date replica (least lag)
  3. Promote it to primary
  4. Repoint other replicas + app to the new primary
  5. (Later) rebuild the old primary as a new replica

Two dangers: split-brain (two nodes both think they're primary — see Consensus & Leader Election) and lost writes (async writes not yet replicated vanish). Fencing and consensus-based election prevent split-brain.

Seeing It in Action

Scenario: Scaling a content site's Postgres, then handling the staleness bug.

# Route reads to replicas, writes to primary
class Database:
    def __init__(self, primary, replicas):
        self.primary = primary
        self.replicas = replicas  # connection pool of read-only followers

    def write(self, query, params):
        return self.primary.execute(query, params)

    def read(self, query, params, consistent=False):
        # Critical/just-wrote reads go to the primary
        if consistent:
            return self.primary.execute(query, params)
        # Everything else load-balances across replicas
        replica = random.choice(self.replicas)
        return replica.execute(query, params)


# Read-your-writes: after a user writes, read consistently for a short window
def update_profile(db, user_id, data):
    db.write("UPDATE profiles SET ... WHERE user_id=%s", (user_id, data))
    session["read_primary_until"] = now() + timedelta(seconds=5)

def get_profile(db, user_id):
    consistent = now() < session.get("read_primary_until", 0)
    return db.read("SELECT * FROM profiles WHERE user_id=%s",
                   (user_id,), consistent=consistent)

What this buys: the site offloads the 90% of traffic that's reads onto cheap replicas, while a small "read from primary for 5 seconds after your own write" rule eliminates the confusing stale-read experience without giving up the scaling benefit.

Interview Questions

1. Q: How do read replicas help scalability, and where's their limit? Hint: They offload read traffic from the primary, scaling reads nearly linearly as you add replicas — ideal for read-heavy workloads (the common case). The limit: every replica must apply every write, so replicas don't scale write throughput or reduce the dataset size. When writes alone saturate the primary, you need sharding.

2. Q: Explain synchronous vs asynchronous replication and the trade-off. Hint: Sync waits for a replica to acknowledge before commit — zero data loss on failover but higher write latency and availability risk if a replica is slow/down. Async commits immediately — low latency, high availability, but risks losing unreplicated writes on primary failure. Semi-sync (wait for one replica) is the common compromise.

3. Q: What is replication lag, and how do you handle a user not seeing their own write? Hint: Lag is the delay before a follower reflects the primary's latest writes. To fix read-your-writes: route the user's reads to the primary for a short window after they write, pin sessions to one replica (monotonic reads), read critical paths from the primary, or wait until the replica catches up to the write's log position (LSN/GTID).

4. Q: What is split-brain during failover, and how do you prevent it? Hint: Two nodes both believe they're primary (e.g., a network partition isolates the old primary that's still alive), leading to divergent/conflicting writes. Prevent with consensus-based leader election (Raft/Paxos), quorums, and fencing (STONITH / epoch tokens) so the old primary is demoted or blocked from writing.

5. Q: When would you choose multi-leader or leaderless replication over single-leader? Hint: Multi-leader for multi-region write locality/offline-capable clients (accept writes near the user), at the cost of conflict resolution. Leaderless (Dynamo/Cassandra) for extreme availability and write-anywhere with quorum reads/writes (R+W>N). Single-leader remains the default when you want simplicity and no write conflicts.

References

• Designing Data-Intensive Applications by Martin Kleppmann — Chapter 5: Replication
• PostgreSQL Replication docs — streaming replication, sync/async modes
• AWS RDS Read Replicas — managed read scaling

Dive Deeper

• GitHub: MySQL High Availability at GitHub (orchestrator) — real failover automation and its pitfalls
• Amazon Aurora paper — decoupling storage from compute for replication
• Jepsen analyses — how replication systems actually behave under partitions