A single region is a single point of failure with very good marketing. It has redundant power, redundant networking and several isolated data centres inside it — and it still goes down, because the things that take out a whole region are rarely physical. A bad configuration push, a control-plane outage, an expired internal certificate, a capacity event during a regional spike.
Multi-region means running your system in more than one geographically separate location so that losing one is survivable. Disaster recovery is the procedure you execute when you lose one. They are not the same thing, and a company can have either without the other: an active-active deployment with no rehearsed runbook, or a single region with excellent backups.
The expensive mistake is starting with the architecture. Start instead with two numbers and one honest sentence about which goal you are actually chasing.
Three goals that get confused
People say "we need multi-region" to mean three unrelated things, and each one produces a different architecture.
Latency. Users in another continent see 200ms of round-trip time before your servers do any work. The fix is to put reads near them — edge caching, regional read replicas — and it barely touches your write path. This is the cheapest goal and the one most often solved adequately by a CDN alone.
Availability. You must survive losing an entire region. This is about redundancy of the write path and of state, which is the hard and expensive part.
Compliance and data residency. Certain users' data must physically stay in a jurisdiction. This is a partitioning problem, not a redundancy one — you shard by region and explicitly forbid the regulated rows from replicating outward. It looks like multi-region and behaves like the opposite of it.
The architectures pull against each other. Latency wants data copied everywhere; residency forbids it. Availability wants automatic failover; residency may make failover to a foreign region illegal. Name your goal before you draw anything, because a design serving all three vaguely serves none of them.
RPO and RTO: the two numbers
Every decision below falls out of these. RPO — recovery point objective. How much data you can afford to lose, measured in time. An RPO of five minutes means that after a disaster, up to five minutes of recent writes may be gone forever.
RTO — recovery time objective. How long you can afford to be down before you are serving traffic again.
They are independent. A nightly backup gives a 24-hour RPO with a 4-hour RTO; synchronous replication gives a near-zero RPO with an RTO of however long failover takes.
| Strategy | RPO | RTO | Relative cost |
|---|---|---|---|
| Backups shipped to another region | Hours | Hours to days | Very low |
| Pilot light (data replicated, compute off) | Minutes | Tens of minutes | Low |
| Warm standby (scaled-down copy running) | Seconds | Minutes | Moderate |
| Hot standby / active-passive | Seconds | Under a minute | High |
| Active-active | Near zero | Near zero | Highest |
Active-passive and active-active
Active-passive keeps one region serving all traffic and a second region receiving replicated data, ready to take over. Reasoning is easy because there is exactly one place writes happen — the leader–follower model from replication, stretched across a continent. The costs are that you pay for capacity that does nothing, and that the passive region is only as good as your last test of it.
Active-active serves live traffic from both regions. You get lower latency for everyone, no idle capacity, and — crucially — you find out continuously whether the second region works, because it is already working. The cost is that you now have two places writes can originate, which is where all the difficulty lives.
The honest summary: active-passive is a correctness-preserving design that fails in a way you have not practiced, and active-active is a permanently harder design that fails in ways you have already survived.
Replicating state across the link
Cross-region replication is the same mechanism you already know, with one number changed: the round trip between regions is 50–150ms instead of under a millisecond. That number changes everything.
Asynchronous replication is the default. Writes commit locally and stream to the other region behind. It costs nothing in write latency, and the replication lag is your RPO — if you are 8 seconds behind and the region dies, you lost 8 seconds of writes. This is why replication lag is not a performance metric; it is a data-loss metric, and it belongs on the alerting dashboard as one.
Synchronous replication waits for the remote region to acknowledge before confirming a write. Your RPO becomes zero and every write pays the round trip. Worse, you have coupled the availability of the two regions: if the remote region is unreachable, writes stall. That is a deliberate CP choice in CAP terms, and it is usually the wrong one across a WAN. The middle ground most systems land on is a quorum that keeps a majority local, so ordinary writes commit at local speed while still surviving the loss of one site.
The hard part: conflicting writes
Two regions accepting writes to the same row will eventually accept conflicting ones — the same user edits a profile from two places, a retry lands in the other region, a partition heals. Someone has to decide which write wins.
Last-write-wins by timestamp is what you get by default, and it silently discards data. Clocks between regions are not perfectly aligned, so "last" is a guess.
Conflict-free replicated data types genuinely solve the problem, but only for data whose merge is mathematically defined — counters, sets, collaborative text. Most business objects are not that.
Application-level merge works when the domain has a rule: carts union, inventory takes the minimum, a status takes the furthest-along value. It requires writing that rule for every conflicting type.
Which is why the common production answer is to sidestep it entirely: route all writes for a given entity to one region. Reads are served locally everywhere; writes for a given user, tenant or shard go to that entity's home region. You accept a latency penalty on a minority of requests in exchange for never merging anything.
Diagram, as text
- EU users leads to EU region
- US users leads to US region
- EU region leads to EU replica (local reads)
- US region leads to US replica (local reads)
- EU region leads to Primary for tenant (writes routed home)
- US region leads to Primary for tenant (writes routed home)
"Active-active for reads, single-writer per entity" is a strong, defensible answer, and it is what most systems that claim to be active-active actually do.
Failover is the part that fails
Steady-state multi-region is not what breaks. The transition is.
DNS is slow and disobedient. Lowering TTL moves traffic in minutes, not seconds, and resolvers and client libraries cache beyond the TTL anyway. Anycast shifts traffic faster because the routing layer does the work, but it gives you less control over which region a user lands in.
Health checks lie in both directions. A check that only tests whether the port is open will happily keep sending traffic to a region that returns errors for every request. A check that is too sensitive will fail over during a transient blip and then fail back — flapping between regions is worse than staying in the broken one.
Split brain. The regions cannot reach each other, so both conclude the other is dead and both start accepting writes. You now have two divergent copies of the truth and a manual reconciliation ahead of you. Preventing it needs an external arbiter or a quorum that only one side can hold, plus fencing so the demoted region genuinely stops writing.
The passive region has never carried the load. Its caches are cold, its connection pools are unwarmed, its autoscaling starts from a scaled-down floor, and its downstream quotas were sized for a fraction of the traffic. A failover that works in a test at 5% traffic can collapse under 100%.
Backups are a different thing entirely
This is the most consequential confusion in the topic. Replication is faithful. It copies your writes, and a DELETE FROM users is a write. So is a bad migration, a ransomware encryption pass, and an application bug that nulls a column. All of them arrive at the replica in milliseconds, perfectly reproduced.
Rehearsal and cost
Run game days: a scheduled exercise where you fail a region deliberately, in production or in an environment close enough to matter, with the on-call team following the written runbook rather than the person who built it. What you are testing is not the technology; it is whether an ordinary engineer at 3am can execute it. Every drill should produce runbook edits.
Be honest about cost, too. Multi-region roughly doubles compute and storage, adds substantial cross-region data transfer charges, multiplies your deployment and observability surface, and makes every engineer reason about replication lag forever. For many systems the correct answer is a single region with genuine multi-zone redundancy, tested backups in a second region, and a documented RTO of a few hours.
Check your understanding
Common mistakes
Conflating latency, availability and residency. Three goals, three architectures, and one design that serves all of them vaguely serves none.
Treating replication as a backup. It copies your deletions perfectly.
Ignoring replication lag as a metric. Lag is your real RPO, not the number written in the design document.
Multi-master by default. Accepting writes in two regions without a conflict rule means last-write-wins is silently discarding customer data.
Failover that has never been executed. An untested runbook is a wish. So is a standby that has never taken production load.
Forgetting the dependencies. Your region failed over; your queue, your identity provider and your third-party payment gateway did not.
Practice
Your product needs RPO of 30 seconds and RTO of 5 minutes. Design the multi-region setup, and state what each requirement rules out.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Convert each number into what it forbids before designing what it allows.
Show a strong answer
Read each number as a prohibition first.
RPO of 30 seconds forbids:
- Nightly backups as the only protection — they permit up to 24 hours of loss.
- Asynchronous replication with lag that is unmonitored or unbounded. Lag is your RPO, so it must be measured continuously and alerted on when it approaches 30 seconds.
It allows asynchronous replication with typical lag of a few seconds, which is important: synchronous cross-region replication would put a round trip of 80–150 ms on every write, and the requirement does not demand it.
RTO of 5 minutes forbids:
- Any procedure requiring a human to be found, woken, and to make a judgement — that alone exceeds five minutes.
- Restoring from backup, which for a large database takes hours.
- DNS-based failover as the primary mechanism, since propagation is uneven and can exceed the budget.
So the design is:
- Warm standby in a second region — the application already running, scaled down but not cold, because starting capacity from zero costs more than five minutes.
- Continuous asynchronous replication, lag monitored as a first-class metric.
- Automated failover with fencing, triggered by health checks rather than by a person, promoting the standby and blocking the old primary from accepting writes.
- Traffic moved at a load balancer or anycast layer, not by changing DNS records.
- Backups still taken separately, because they cover the corruption case that replication cannot.
The one thing that makes this real: rehearse the failover on a schedule. An untested procedure has an RTO of "unknown", whatever the document says.
Active-active across two regions means both accept writes. Give two concrete conflict scenarios and how you would resolve each.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask whether the two writes can be merged, or whether one must win.
Show a strong answer
Scenario one — the same user updates their profile from two devices routed to different regions. Both writes succeed locally; replication then carries each to the other side.
- Resolution: last-write-wins on a per-field basis, using a timestamp with a deterministic tiebreaker such as region ID. Merging per field rather than per record means a name change in one region and a phone change in the other both survive, instead of one overwriting the whole record.
- Why acceptable: losing one of two near-simultaneous edits by the same person is a minor annoyance, and they will notice and redo it.
Scenario two — the last item of stock is sold in both regions simultaneously. Both succeed; the merged state shows two orders for one item.
- Resolution: last-write-wins is unacceptable here — it does not undo the second sale, it just records a stock count that is wrong. Someone has bought something that does not exist.
- The honest options are to make inventory single-writer — one region owns each SKU, and writes for it are routed there regardless of where the user is — or to accept oversell and compensate with an apology and a refund, which many retailers deliberately choose.
The general rule this illustrates: active-active is safe for data that can converge and unsafe for anything that allocates a finite resource. Partition by ownership rather than by user location for those cases, and accept the cross-region latency on the small subset of writes that genuinely require it.
Key takeaways
- Decide first whether you are buying latency, availability or compliance — they want different, sometimes opposing architectures.
- RPO and RTO are business decisions that then determine the tier, from cross-region backups up to active-active.
- Asynchronous replication's lag is your data-loss window; synchronous replication buys RPO zero by coupling the availability of both regions.
- Conflicting writes have no free solution, so most real systems use active-active reads with a single write region per entity.
- Failover is the thing that fails — DNS caching, lying health checks, split brain, cold standbys and untested runbooks.
- Replication is not a backup. Only immutable, separately-credentialled, regularly-restored backups survive a logical mistake.
- Rehearse with game days, and accept that for many systems a single region with multi-zone redundancy and tested backups is the correct, honest answer.
Further reading
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 26 — data integrity, and why replication is not backup.
- James Corbett et al., "Spanner: Google's Globally-Distributed Database" (OSDI 2012) — synchronous global replication and its latency cost.
- Marc Shapiro et al., "Conflict-free Replicated Data Types" (SSS 2011) — convergence for active-active data.
- Werner Vogels, "Eventually Consistent" (ACM Queue, 2008).
- Betsy Beyer et al., The Site Reliability Workbook (O'Reilly, 2018), chapter 12 — testing disaster recovery rather than documenting it.