Skip to content
All problems

Regional failover for a train ticket status lookup

EasyAbout 30 minutes

Before you start:Domain Name System (DNS)

Your attempt

A complete grade of 70 or more solves it.

A train ticketing platform lets a passenger check whether their booked ticket has been confirmed, put on the reservation against-cancellation list, or is still waitlisted, by entering their ticket number. It is a small, read-only lookup, but it is asked constantly — before a journey, and obsessively in the days before a festival, when a waitlisted passenger refreshes the page hoping their number has moved. You own the lookup service's regional layout: how it stays up when a whole region's fleet goes down.

The service runs as two complete, independent copies, one in each of two regions, both reading the same reservation data. Passengers reach it through a single DNS name. Normally every lookup goes to the primary region; the standby sits ready behind it, not serving live traffic but not idle either — it has to be able to take over the country's entire lookup load the moment it is needed, because a region going down and a festival rush arriving tend to happen in the same unlucky week.

What the lookup does

  • A passenger looks up their ticket's status by ticket number, and the answer reflects the reservation system's recorded state, at most a moment behind it.
  • Passengers reach the service through one DNS name; while the primary region's fleet answers its health checks, every lookup goes to it.
  • When the primary region's fleet stops answering, DNS moves new lookups to the standby region on its own, without a passenger refreshing manually or the operator paging anyone at 2 a.m.

The numbers

  • About 34,560,000 status lookups happen a day nationally. The evening peak, when most passengers travelling the next day check before bed, runs at about five times the daily average. In the days before a major festival, the rate rises to about 2.5 times the evening peak and stays there through most of the day rather than for just a burst.
  • A status lookup costs the service about 3 ms of CPU. Set read_cpu_ms on your service to match.
  • DNS health checks run every ten seconds, and the record's TTL decides how long a passenger's own answer stays cached after it stops being true.

What good looks like

A lookup answers within 150 ms at p99, and with the primary region healthy, at most 1 lookup in 1,000 fails — the festival rush included. If the primary region's fleet fails outright, at most 1 lookup in 5 fails over the first 50 seconds, and lookups are back under 1% failing a second within 20 seconds.

Where to start

Do the estimation step first: work out how many instances one region needs to carry the festival rush by itself, then remember that number applies to the standby too, not half of it — a standby sized for "normal" traffic looks fine until the day it actually has to answer for the whole country. Then set DNS to fail over rather than simply answering with the first target, and work out what its health-check interval and TTL together bound about how long a passenger keeps calling a region that is already down — a lookup this cheap to re-resolve can afford a very short TTL to keep that bound tight.

Out of scope: the reservation and booking flow itself, the database's own cross-region replication, and how a passenger's ticket number is validated before the lookup runs.

Requirements

Functional

  • A passenger looks up their ticket's status by ticket number; the answer reflects the reservation system's recorded state, at most a moment behind it.
  • Passengers reach the status service through one DNS name; while the primary region's fleet is healthy, every lookup goes to it.
  • When the primary region's fleet stops answering, DNS moves new lookups to the standby region on its own, without a passenger having to do anything.

Non-functional

  • Both regions read the same reservation data, a managed table already redundant across zones; that table's own failover is out of scope here.
  • The standby region is not a cold spare — a region failure and a festival travel rush have a habit of landing in the same week, so the standby has to carry the full national load on its own if it must.
  • Passengers cache almost nothing on their end; every lookup is a fresh request, so DNS's own answer is what has to move them off a dead region.

Estimation

Before you draw, estimate these. The workspace checks each answer against its band as you type.

  1. Status lookups a second at the evening peak (lookups/s)
  2. Status lookups a second during a festival rush (lookups/s)
  3. Service instances each region needs to carry the festival rush alone (instances)
  4. Longest a passenger keeps calling a dead region, with a zero-second TTL (s)

Tests

2 sample tests you can read and run

  • DNS is set to fail over, not just to answer the first target (Functionality)
  • Lookups meet the SLO at the evening peak (Performance)

2 hidden tests, graded when you submit

  • Scalability1
  • Reliability1

What the design must meet

Service levels

  • A status lookup answers within 150 ms at p99.
  • With the primary region healthy, at most 1 lookup in 1,000 fails.
  • Over the first 50 seconds after the primary region's fleet goes down, at most 1 lookup in 5 fails, most of them before DNS finishes moving everyone to the standby.
  • Within 20 seconds of the primary region's fleet failing, lookups succeed again at under 1% failing a second.

Load

  • The evening peak, when most passengers travelling the next day check before bed, runs at about five times the daily average of status lookups nationally.
  • In the days before a major festival, waitlisted passengers check obsessively and the rate rises to about 2.5 times the evening peak through most of the day.

How an attempt runs

  1. RequirementsWhat the service must do, and the numbers it must meet
  2. EstimateSize the load before you draw
  3. DesignBuild it on the canvas and run traffic through it
  4. TestRun the sample tests in your browser
  5. SubmitThe hidden tests grade the design