Your application servers scale beautifully. Add a machine, handle more traffic. But they all talk to one database, and that database has two problems no amount of application servers can fix.
It is a single point of failure. When it dies, everything stops. Not degraded — stopped.
It is a read bottleneck. Most applications read far more than they write, often 100:1. One machine can only serve so many reads.
Replication addresses both by keeping copies of your data on other machines.
Leader and followers
The dominant arrangement, and the one you should assume unless told otherwise.
One node is the leader (also called primary or master). All writes go to it. Every change is streamed to one or more followers (replicas, secondaries), which apply the same changes in the same order.
- App servers send writes to the leader.
- Reads are spread across two replicas that stream changes from the leader.
Describe this diagram
Components
- app (Client)
- service (Service)
- db (PostgreSQL)
Connections
- app sends to service
- service sends to db
Two payoffs immediately:
Read scaling. Spread reads across followers. With a 100:1 read-write ratio, three followers roughly quadruple read capacity while the leader handles the same modest write load.
Failover. If the leader dies, promote a follower. Recovery becomes minutes or seconds instead of restoring from backup.
There is also a third benefit that gets forgotten: you can point analytics and reporting at a follower, so a colleague's enormous query cannot slow down customer traffic.
Synchronous or asynchronous: the real decision
When a write arrives at the leader, when does it tell the client "done"?
Synchronous — wait until at least one follower confirms it has the change, then acknowledge.
- Guarantees no data loss if the leader dies immediately afterwards.
- Every write pays the network round trip to the follower.
- If that follower is slow or unreachable, writes block. Your availability now depends on the follower too.
Asynchronous — acknowledge as soon as the leader has it, and stream to followers in the background.
- Fast writes; leader availability unaffected by follower health.
- If the leader dies before the change reaches any follower, that write is lost. Permanently.
Most systems run asynchronous, because the performance difference is large and the loss window is normally milliseconds. But be clear that you have chosen to accept losing recent writes in a crash.
Replication lag, and the bug it causes
Followers are always slightly behind. Usually milliseconds; under load or long transactions, seconds or worse.
This produces the single most common bug in replicated systems, and it is worth internalising because it will bite you:
- A user updates their profile name. The write goes to the leader.
- The page reloads and reads from a follower.
- The follower has not received the change yet.
- The user sees their old name and concludes the save failed.
They save again. And again. Nothing is broken — the system is behaving exactly as designed — but the user experience is terrible.
This is a violation of read-your-own-writes consistency: whatever else may be stale, you should always see your own changes.
The standard fixes:
- Read from the leader for a short window after a user writes — simple, effective, and the most common approach in practice.
- Route a user's reads to the leader for data they own, and use followers for everything else.
- Track a write position and require the follower to have caught up to it before serving that user.
- Serve it from the client — the browser already knows what it just submitted.
What replication does not solve
This distinction is important, and interviewers push on it.
Replication copies all the data to every node. It scales reads and provides redundancy.
It does not scale writes — every write still goes to the leader, so write throughput is capped by one machine. And it does not help with data volume: if your dataset is 10TB, every replica holds 10TB.
When you outgrow either of those, you need partitioning, which is the next lesson. The two are complementary and production systems use both: shard the data across machines, and replicate each shard for safety.
Failover, and the danger in it
When the leader dies, a follower is promoted. Automatic failover sounds obviously good and carries real risks.
Data loss. With asynchronous replication, writes the old leader accepted but had not yet sent are gone. When the old leader returns, it has changes the new leader never saw — usually discarded.
Split brain. If the old leader was not actually dead, only unreachable, you now have two nodes accepting writes. They diverge, and reconciling is painful. This is the CAP partition scenario made concrete: systems use fencing or quorum to ensure only one leader can be active.
Cascading failure. The new leader inherits the full write load, plus a stampede of reconnecting clients, and may itself fall over.
Multi-leader and leaderless, briefly
Multi-leader allows writes at several nodes — useful across regions so users write locally. The cost is write conflicts: two regions edit the same record simultaneously and someone must decide who wins. Conflict resolution is genuinely hard, which is why this is uncommon outside specific cases like collaborative editing.
Leaderless (Dynamo-style, as in Cassandra) sends writes to several nodes and reads from several, using quorums: if writes go to W nodes, reads to R nodes, and W + R > N, at least one node you read has the latest value. This is the tunable dial between consistency and availability mentioned in the CAP lesson.
Check your understanding
Common mistakes
Assuming replicas are current. They are eventually consistent by default. Any code assuming a read reflects a just-completed write is wrong on a follower.
Sending writes to a follower. Followers are read-only. Frameworks that route by connection pool will fail confusingly if misconfigured.
Replicating instead of sharding. If your problem is write throughput or data volume, more replicas will not help — each holds everything and only the leader accepts writes.
No monitoring of lag. Replication lag is a key metric. Rising lag predicts read-your-own-writes bugs and a failover that would lose more data.
Forgetting replicas cost money. Three replicas means three times the storage and the write work applied three times.
Practice
Your leader fails. The most up-to-date follower is 4 seconds behind. Describe what you lose by promoting it, what happens if the old leader comes back, and how you would prepare for this in advance.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask what was acknowledged to clients in those 4 seconds, and where those writes exist now.
Show a strong answer
What promoting costs you. Every write acknowledged in the last 4 seconds exists only on the dead leader. Clients were told those writes succeeded. Promoting the follower makes them vanish — an order confirmed by email that no longer exists in the database. This is not a bug in failover; it is the price of asynchronous replication, paid at the worst possible moment.
When the old leader returns, it holds writes the new leader never saw, and it still believes it is the leader. If anything is still routing to it, you now have two leaders accepting conflicting writes — split brain — and afterwards no automatic way to decide which version of a row is correct. The standard defence is fencing: the promoted leader takes a monotonically increasing term number, and storage refuses writes carrying an older term, so the returning leader is locked out even if it is still trying.
How to prepare:
- Decide the trade in advance, per dataset. Semi-synchronous replication for anything where a lost acknowledged write is unacceptable — payments, orders — and asynchronous where 4 seconds of loss is survivable.
- Make failover automatic but fenced, because a human deciding takes longer than the data loss you are arguing about.
- Know your replication lag as a monitored number, not a hope. Lag is your recovery point objective, whether or not anyone wrote it down.
- Rehearse it. A failover path that has never been executed is a plan, not a capability.
A read replica is added to reduce load on the leader. Load on the leader does not fall. Give three plausible reasons.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Adding a replica only helps if something actually reads from it.
Show a strong answer
- Nothing was routed to it. The application still uses one connection string. Adding a replica does not redirect traffic; something must decide, per query, that this read may go elsewhere. This is the most common answer and the least interesting one — check it first.
- The reads were never the problem. If the leader was write-bound — index maintenance, lock contention, transaction commits — then removing reads changes little. Worse, the leader now also does the work of shipping changes to the replica, so a write-bound leader can get slower when you add one.
- The queries that matter cannot safely use a replica. Anything on a read-your-own-writes path, or inside a transaction that also writes, must go to the leader. If most of the read volume is on those paths, the replica can only serve the reporting queries that were never the load.
How to tell them apart: look at where the leader's time goes before assuming. If it is dominated by commits and index updates, no number of read replicas helps and the answer is partitioning. If it is dominated by reads that nothing is routing away, the fix is in the application's routing, not the database.
Key takeaways
- Replication solves read scaling and redundancy by keeping copies; writes still all go to the leader.
- Asynchronous replication is fast but can lose recent writes in a crash; synchronous is safe but couples your writes to follower health. Semi-synchronous is the common compromise.
- Replication lag causes read-your-own-writes bugs — the user sees stale data after saving. Route their reads to the leader briefly after a write.
- Watch also for monotonic read violations, where refreshing hits a further-behind replica and data appears to go backwards.
- Replication does not scale writes or reduce data volume — that is partitioning, and real systems use both.
- Failover carries risk: lost writes, split brain, and a new leader inheriting full load.
Next: when one machine can no longer hold the data or absorb the writes, you must split the data itself.
Further reading
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 5 — leader-based, multi-leader and leaderless replication.
- Jim Gray, Pat Helland, Patrick O'Neil and Dennis Shasha, "The Dangers of Replication and a Solution" (SIGMOD 1996) — why multi-leader replication scales badly.
- Diego Ongaro and John Ousterhout, "In Search of an Understandable Consensus Algorithm" (USENIX ATC 2014) — Raft, including terms and fencing.
- Werner Vogels, "Eventually Consistent" (ACM Queue, 2008) — replication lag framed as a product decision.
- Bruce Momjian, "Postgres Replication Internals" (conference tutorial, 2018) — how log shipping actually works.