Single point of failure (SPOF)

For this lesson, we'll learn to sweep a diagram box by box, find the SPOFs nobody drew (DNS, certificates, config pushes, a control plane, a person), spot redundancy that only looks redundant, and fix each one with the cheapest option that's good enough. Interviewers ask "are there any single points of failure here?" in nearly every round, because it tests whether you can look at your own design critically and in a systematic way, rather than defending it.

01Picture it first: one box, whole site dark

Here's the smallest design anyone draws: a load balancer, an app server and a database. Every box in it is a SPOF. The fix isn't "make everything redundant". It's a specific, cheap change per box, and the database needs a different fix from the other two, because it holds state.

Notice that the "after" picture is barely more complex. Most SPOF fixes in an interview are one sentence and one extra box. The hard part is finding them, and the boxes that aren't on the diagram.

02The sweep: a method you can run out loud

Azure's Well-Architected guidance calls the formal version failure mode analysis: break the workload into components, follow each critical flow, and for every component ask how it can fail and what the blast radius is. In an interview I run a lighter version in about two minutes:

1. Pick the critical flows. "Place an order" and "view the product page", not every endpoint.
2. Walk each flow from the user to the disk. DNS, CDN, load balancer, gateway, services, caches, queues, databases, third-party APIs.
3. For each box, ask three questions. If it dies, what breaks? If it's slow, what breaks? Is there exactly one of it?
4. Rate it. How likely is the failure, and how many users does it hit? A one-in-ten-years event on an internal report can stay.
5. Fix it or accept it, out loud. "I'll add a standby" or "I'm accepting this one because...".

Asking about reads and writes separately pays off: a database primary is often a SPOF for writes while replicas keep reads alive.

Here's how I'd walk it:

Notice that the two worst items, DNS and the certificate, aren't boxes on the diagram at all. And notice the "No, but..." row: six servers sounds redundant until you ask where they live.

03Hidden SPOFs: the boxes nobody drew

The obvious SPOFs (one database, one load balancer, one cache node) are the ones interviewers expect you to catch. The ones that separate a senior answer are shared things that sit underneath every box:

Two real incidents show how these chain together.

The lesson from both: your recovery path and your status page are part of the system. Sweep them too.

04Redundant on paper, one failure in real life

Two boxes on a whiteboard aren't automatically two failure domains. If both servers hang off one power feed, one top-of-rack switch or one zone, a single fault takes both. The same goes for things you can't see on any diagram: the same config, the same deploy, the same OS image, the same bug.

05Fix it, cheapest first

Once you've found a SPOF, go up this ladder only as far as the requirements push you:

A few defaults I use:

• N+1 vs N+2. N+1 means one spare beyond the capacity you need. N+2 means you can lose one instance while another is down for a deploy or maintenance. For the critical path I'd say N+2, or N+1 per zone across three zones.
• Stateless first. Moving state out of a box (sessions into signed tokens or a replicated store) turns a hard SPOF into an easy one.
• Accept some on purpose. An internal admin tool, a nightly batch job that can rerun, a single-region analytics pipeline. Saying "this is a SPOF and I'm accepting it because the impact is small" is a strong signal, not a weak one.

Azure's guidance on designing for redundancy lands in the same place: size redundancy to how critical each flow is, prefer managed services that handle replication and failover for you, and keep enough spare capacity that the survivors aren't overloaded when a copy dies. Two nodes each running at 60% can't absorb each other.

06How it shows up in design problems

Every design round has a SPOF moment. These are the ones where it's the heart of the problem:

• Design a distributed lock: the lock service itself is the SPOF everyone forgets. The answer is a quorum (Consensus algorithms, ZooKeeper and etcd) plus leases so a dead holder doesn't block forever.
• Design a job scheduler: one scheduler process is a SPOF; several running at once fire jobs twice. You need leader election and idempotent jobs.
• Design a distributed cache: one node per shard is a SPOF for that key range, and a hot key can make one node the bottleneck. Replicas per shard and data replication fix the first.
• Design a payment system: the external payment provider is a dependency you can't make redundant yourself; some designs add a second provider for exactly this reason.

For the infrastructure pieces, see Load balancing and DNS. For how a dead node gets noticed in the first place, see the health checks pattern.

07How to say it in the interview

When I ask "any single points of failure?", I'm listening for a systematic sweep rather than a guess, at least one hidden SPOF that isn't on the board, and a fix sized to the requirement. Candidates who say "I'd accept this one, here's why" usually impress me more than those who make everything redundant.