Fault tolerance

For this lesson, we'll walk through the toolkit: redundancy, timeouts, retries with backoff and jitter, idempotency, circuit breakers, bulkheads, graceful degradation and load shedding, and when each one earns its place. Interviewers ask "what happens when service X is slow or down?" in almost every design round, because it separates people who've drawn boxes from people who've been paged at 3 a.m.

01Picture it first: which fault are you tolerating?

Faults come in a few shapes, and each one needs a different defence. Name the shape before you name the pattern.

Slow is the hardest. A crashed node is easy to spot and route around. A slow one still accepts connections, still answers health checks, and quietly holds every caller's threads while it thinks. Most of the toolkit below exists because of slowness.

02Redundancy: the baseline

Run more than one copy of everything that matters, in places that don't fail together: separate machines, racks, zones. Stateless services sit behind a load balancer; stateful ones replicate and fail over. This is the foundation, and it's cheap to say, but it only helps if the copies are independent. Two replicas on the same rack or the same bad config file are one failure waiting to happen. Finding those hidden shared parts is the job of Single point of failure; the maths of how much redundancy buys you lives in Availability.

03Timeouts: never wait forever

Every call across a network, and ideally every call across processes, needs a timeout. Without one, a hung dependency holds your thread, your connection and your caller's thread, all the way up the stack.

Picking the value is the hard part. Amazon's Builders' Library describes a practical rule: choose an acceptable rate of false timeouts, such as 0.1%, and set the timeout at the matching latency percentile of the dependency, here its p99.9. Too high and the timeout protects nothing; too low and healthy-but-slow requests get retried, adding load exactly when the dependency is struggling.

Two refinements show depth in an interview:

• Budgets, not fixed numbers. If the user-facing request has 2 seconds, and 1.4 seconds are already gone, the next call gets at most 0.6 seconds. Google's SRE book calls this deadline propagation: pass the remaining time down the stack so no one works on a request the caller already gave up on.
• Inner timeouts shorter than outer ones. If the gateway times out at 3 seconds but the service behind it waits 10 seconds on the database, the service keeps working for 7 seconds on an answer nobody will read.

04Retries with backoff and jitter

Many faults are transient: a dropped packet, a node restarting, a brief leader election. Retrying fixes those. But retries are also how small incidents become large ones, because every retry is extra load sent to something that's already failing.

The answer is 3 x 3 x 3 x 3 x 3 = 243. Each layer multiplies the attempts of the layer above. This exact example appears in Amazon's guidance, which concludes that a database under that pressure is unlikely ever to recover. So the rules for retries are:

1. Retry at one layer, usually the one closest to the user or the one that owns the operation. Others fail fast.
2. Cap the attempts (2 or 3), and cap them globally too: a retry budget stops retrying when too many recent calls were retries. Google's SRE book suggests a per-process limit, for example 60 retries a minute. Amazon's SDK limits retries locally with a token bucket.
3. Back off exponentially, doubling the wait each time, so a struggling service gets breathing room.
4. Add jitter, a random part of the wait, so thousands of clients that failed together don't retry together.
5. Only retry what's safe. Reads are usually safe. Anything with side effects, like charging a card, needs an idempotency key so the second attempt doesn't charge twice.

A common form is "full jitter": wait a random time between zero and the current backoff cap. Notice in the figure that the very first burst is identical in both cases. Jitter can't undo the original failure; it stops you from recreating it on every retry. For the patterns side by side, see Distributed systems patterns, and for how these failures look in production, Backend failure modes.

05Circuit breakers: stop calling what's broken

Retries assume the fault is brief. When it isn't, a circuit breaker watches the error rate to a dependency and, past a threshold, fails calls immediately for a cool-off period, then lets a few trial calls through to see if it has recovered. Callers get a fast error they can handle, and the sick dependency gets room to recover.

Breakers aren't free. Amazon's Builders' Library points out that they add modal behaviour that's hard to test and can lengthen recovery, which is why Amazon often prefers limiting retries with a token bucket. My default: bounded retries everywhere, breakers on dependencies that fail for long stretches or are expensive to wait on. The state machine is covered in Circuit breaker.

06Bulkheads: contain the blast radius

Ships are split into watertight compartments so one breach doesn't sink the whole hull. In software, a bulkhead gives each dependency, tenant or feature its own slice of resources: a thread pool, a connection pool, a set of servers, or a whole cell.

The same idea scales up. Cells run complete, independent copies of a service, each serving a slice of users, so a bad deploy or a poison request hurts one cell. Shuffle sharding goes further by giving each customer a random small set of workers.

07Graceful degradation and load shedding

When something breaks, a smaller answer usually beats no answer.

• Graceful degradation means deciding in advance what each feature does when its dependency is down. Recommendations fall back to a cached "popular items" list. Reviews disappear from the product page while checkout keeps working. A feed shows slightly stale posts from a cache.
• Load shedding means refusing some work early when you're over capacity, so the work you accept still completes in time. Reject with a quick 429 or 503 at the edge, drop low-priority traffic (batch jobs, prefetches) first, and drop requests that have already waited longer than their deadline. Google's SRE book describes the goal well: keep the server out of overload while it still does as much useful work as it can, for example by returning 503 once too many requests are in flight. Rate limiting and message queues are the neighbouring tools: one caps what a client may send, the other absorbs bursts to process later.

08Prove it: inject failures on purpose

You don't know a system tolerates a fault until you've watched it do so. Chaos engineering turns that into a habit: define steady state as a measurable output (orders per minute, error rate), predict it'll hold, inject a real-world fault such as a killed server or a severed connection, and look for a difference. Netflix's open-source Chaos Monkey does the simplest version, randomly terminating production instances so every team has to build services that survive it. Start small: one instance, off-peak, with a way to stop the experiment.

09How it shows up in design problems

Every design problem has at least one "what if this is slow?" moment. Three where it decides the design:

• Design a payment system: retries to the payment provider must be idempotent, timeouts must be explicit about "unknown" outcomes, and you reconcile instead of guessing.
• Design a notification system: each channel (email, SMS, push) gets its own queue and worker pool, so a slow SMS provider doesn't delay push notifications, with retries and backoff per channel.
• Design a flash sale: load shedding and queueing at the edge protect inventory, and non-critical pages degrade so the purchase path keeps its capacity.

At this point, you should ask your interviewer which features are critical. The answer tells you where to spend redundancy and where a fallback is good enough.

10How to say it in the interview

When I ask "what if this dependency is slow?", I'm listening for timeouts with a reason behind the number, bounded retries that are safe to repeat, isolation so the problem stays local, and a plan for what the user sees. Candidates who mention retry amplification or jitter unprompted usually have real production experience.