Failover

For this lesson, we'll focus on the switch itself: the two main topologies, how a system decides something is dead, how traffic finds the new owner, and the two ways it goes wrong (losing data and ending up with two primaries). Interviewers love asking "what happens when your primary database dies?" because a one-word answer ("replicas!") is easy, and a good answer has to cover timing, data and correctness at once.

01Picture it first: failover is a timeline

Most people picture failover as one event. In practice it's a short sequence, and each step costs time, so I always draw it as a timeline.

Notice three things:

• Detection is deliberately slow. You wait for several missed checks because declaring a healthy node dead is expensive. That wait is part of the outage.
• Promotion needs a decision. Something has to agree that the old primary is gone and pick one successor. Done carelessly, this is how you get two primaries.
• The switch depends on clients. Changing a DNS record takes seconds. Clients that cached the old address keep using it until their cache expires.

02Active-passive vs active-active

There are two basic shapes. Pick based on how much you'll pay to shorten recovery, and whether your data can live with more than one writer.

For stateless services, active-active is the default and failover is almost free: the load balancer drops an instance that fails its health check and the rest absorb the load, if you left headroom. For databases, single-writer active-passive is the usual answer, because two writers on the same rows need conflict resolution. When I hear "active-active database", I ask how conflicts are resolved, and the answer had better be specific.

03Detecting failure without crying wolf

A machine can't tell "dead" from "slow" or "unreachable from where I'm standing". All it sees is silence. Every detection scheme is a bet on how long to wait.

• Heartbeats. The node sends "I'm alive" every interval; miss enough and it's presumed dead. We cover the mechanics in Heartbeats.
• Health checks. Something outside probes the node, ideally with a request that touches its real dependencies, not just "process is up".
• Consecutive-failure thresholds. One failed probe proves little. Requiring several in a row trades detection speed for fewer false alarms.
• Many observers and a quorum. One checker behind a bad switch will see everyone as dead. Several checkers in different places, with a majority rule, filter that out.

Amazon Route 53 documents all four. Its health checkers in several locations probe each endpoint every 30 seconds, or every 10 seconds for an extra charge. You set a failure threshold, the number of consecutive checks that must fail before the status flips. And it only marks an endpoint unhealthy if 18% or fewer of its checkers report it healthy, so a network problem near one checker doesn't fail you over.

04Switching traffic to the new owner

Once there's a new primary, clients need to find it. Four common mechanisms:

The DNS row matters because many managed databases fail over that way. AWS documents that an RDS Multi-AZ failover changes the DNS record of the instance to point at the standby, and recommends configuring the JVM's DNS cache TTL to no more than 60 seconds, because some Java configurations cache a lookup until the JVM restarts. That's a failover that worked on the server and never reached the app. DNS basics are in Domain Name System.

05The two numbers: RPO and RTO

Every failover answer should end with two numbers.

• RPO (recovery point objective): how much recent data you can afford to lose. It's set by replication. A synchronous standby acknowledges each commit before the client hears "ok", so a committed write is never lost, at the cost of extra latency on every write (AWS notes that Multi-AZ can add write and commit latency for this reason). An asynchronous replica lags behind, and whatever it hadn't received when the primary died is gone or stranded.
• RTO (recovery time objective): how long users wait. It's detection plus decision plus switch, as the timeline showed.

Data replication covers the replication modes, and Databases for system design interviews shows replication lag in practice. Disaster recovery covers setting RPO and RTO targets for a whole region.

Here's how I'd work it out loud. All of these are assumptions for the exercise, so measure your own.

Notice which number drives the design. The DNS TTL is the largest piece of RTO, so lowering it, or using a proxy that clients don't cache, helps more than speeding up promotion. And the RPO says a payment system can't use async replication for its ledger, because 1,000 acknowledged payments might disappear.

06Split brain: two primaries at once

Here's the failure that turns a short outage into a long one. A partition cuts the primary off from the cluster but not from all its clients. The cluster presumes it dead and promotes a replica, and now two nodes both accept writes.

The defences stack:

1. Decide by majority. Use an odd number of voters (3 or 5) and require a majority to promote. A minority side of a partition can't elect anyone. This is what Consensus algorithms like Raft give you (see also leader election).
2. Fence the old primary. Make it physically unable to write: cut its power or network, revoke its storage access, or let its lease expire before promoting anyone. The PostgreSQL docs call the blunt version STONITH ("Shoot The Other Node In The Head") and warn that without some such mechanism you can end up with two primaries and, eventually, data loss.
3. Number the leaders. Each promotion gets a higher term or epoch, and storage or downstream services reject requests with an older one. A primary waking up from a long pause then finds its writes refused. Distributed locking uses the same idea under the name fencing tokens.

07Failing back, and testing all of it

After a failover you're running on one copy with no spare (the PostgreSQL docs call it a degenerate state). Rebuild a standby straight away, rejoining the old primary as a replica only after discarding writes the new primary never saw. Whether you then fail back is a choice, but make it a planned, practised operation.

A failover that has never been exercised is a hope, not a plan. Trigger it on a schedule in a quiet window and measure the RTO you actually get. Fault tolerance covers chaos testing more broadly.

08How it shows up in design problems

Failover matters wherever state can't be rebuilt in seconds.

• Design a payment system: the ledger needs RPO zero, so a synchronous standby in another zone, fencing, and idempotent retries from clients while the switch happens.
• Design a key-value store: leader-based partitions need leader election per partition, plus a story for writes the old leader accepted but never replicated.
• Design a distributed lock: the lock service itself fails over, so the interesting question is how a client that held a lock on the old leader is stopped, which is fencing again.

At this point, you should ask your interviewer how much data loss is acceptable. "Zero" and "a few seconds" lead to different designs, and asking shows you know that.

09How to say it in the interview

When I ask "what happens when the primary dies?", I'm listening for four things: detection without false alarms, who decides and how you prevent two primaries, how clients find the new one, and your RPO and RTO.