Engineering Atlasby Abhishek Goyal
Course contents

Reliability patterns cheat sheet

A one-page cheat sheet of reliability patterns for system design interviews, grouped by the failure each contains: timeouts, retries with jitter, circuit breakers, bulkheads, rate limiting, load shedding, queues, idempotency, sagas, outbox, cells, failover, replication and backups.

FreeOften askedLesson10 min readBeginner → SeniorReviewed Oct 2026
On this page
  1. Picture it first: where each pattern sits
  2. A dependency is slow or down
  3. Too much load
  4. A step fails halfway
  5. One failure takes everything down
  6. We lost data or a region
  7. When patterns fight each other
  8. Pick three
  9. How to say it in the interview

🧰 What is this cheat sheet?

It's every reliability pattern you're likely to name in a design round, on one page, sorted by the failure it contains. Think of a first-aid kit where each pouch is labelled by injury: "burns", "cuts", "sprains". You name what went wrong, open that pouch, and take the one or two things that fit.

For this page, we'll skip deep explanations. Each pattern gets one line: the failure it contains, when to use it, what it costs, and where we cover it properly. Interviewers don't reward a long list of pattern names. They reward picking the three that matter for this design and saying why.

Below the line (out of scope)

Name the failure first, then the pattern

I never start with a pattern. I start with the failure: a dependency is slow or down, there's too much load, a multi-step operation fails halfway, one failure spreads to everything, or we lose data or a region. Each failure has a short list of patterns: timeouts, jittered retries and a circuit breaker for a slow dependency; rate limits, load shedding and queues for overload; idempotency keys and sagas for half-finished work; redundancy and cells for blast radius; replicas plus backups for data. Then I pick the two or three that matter most for this system and explain what each one costs.

What you’ll learn

  • Start from the failure (slow dependency, overload, half-done work, blast radius, data loss) and only then pick a pattern
  • Place each pattern on the request path: client, gateway, service, data, queue
  • Name the cost or gotcha of every pattern you propose, not only the benefit
  • Spot patterns that fight each other, such as retries at every layer multiplying load
  • Pick the three patterns a payment API, a news feed or ride matching needs, and say why

01Picture it first: where each pattern sits

Most patterns live at a specific hop on the request path. Place them here and you can place them on any whiteboard.

Fig. 1

Where each reliability pattern sits on a request path

Where each reliability pattern sits on a request pathA client calls a load balancer or gateway, which calls a service. The service calls an external dependency, writes to a database, and hands slow work to a queue drained by workers. Pattern tags sit on each hop: retries with backoff and jitter plus idempotency keys at the client; rate limiting, load shedding and health checks at the gateway; timeouts, circuit breaker, bulkhead and fallback in the service; replication, failover and backups at the database; load leveling, competing consumers and the outbox at the queue. The gateway, service, database and queue form one cell, so a failure stays inside it; more cells serve other users, and the dependency sits outside.One cell (deployment stamp)callwriteslow workClientapp, browserGateway / LBServiceour codeDependencye.g. payment providerDatabaseprimary + replicaQueue +workersCell 2, cell 3...same stack, other users;a bad cell hurts onlyits own shareretry + backoff+ jitteridempotency keyrate limitingload sheddinghealth checkstimeoutscircuit breakerbulkhead poolsfallbackoutside our cell:slow or down any timereplication + failoverbackups (RPO / RTO)load levelingcompeting consumers, outbox
Each hop has its own defences. Clients retry safely, the gateway protects capacity, the service protects itself from its dependencies, and data gets copies plus backups. The cell boundary keeps a failure inside one slice of users.

Notice that the service, the box we own, carries the most patterns: it has to survive everything around it misbehaving. The dependency sits outside our cell because we can't fix it, only defend against it.

02A dependency is slow or down

The failure interviewers ask about most. Slow is worse than down: a dead dependency fails fast, a slow one ties up your threads.

Pattern Failure it contains Use when Cost / gotcha Our page
Timeout A slow call holding a thread forever Every network call, no exceptions Must be shorter than the caller's own deadline, or it protects nothing Fault tolerance
Retry with backoff and jitter A brief blip: dropped connection, a node restarting Transient errors on calls that are safe to repeat Multiplies load during a real outage; cap attempts, retry at one layer Fault tolerance
Circuit breaker Hammering a dependency that's already down Failures that last seconds to minutes Needs a fallback for the open state, and tuning so it doesn't flap Circuit breaker
Fallback (graceful degradation) A non-critical feature breaking the whole page There's an acceptable worse answer: cached, default or hidden Fallback code that never runs quietly rots; exercise it Fault tolerance
Bulkhead One slow dependency using up every thread or connection Several dependencies share one pool Separate pools leave some capacity idle Fault tolerance
Cache-aside Reads failing when the database has a bad minute Read-heavy data that can be a little stale Stale reads; a cold cache after a restart floods the database Caching strategies

03Too much load

The goal is to say no to some work early and deliberately, so you can say yes to the rest.

Pattern Failure it contains Use when Cost / gotcha Our page
Rate limiting (throttling) One client or tenant using more than its share Public APIs, multi-tenant systems Needs a shared counter across servers Rate limiting
Load shedding Every user getting slow when the service is past capacity Load can jump faster than you can add servers Decide what to drop first; dropped users see errors Fault tolerance
Queue-based load leveling A spike crushing a slower backend The work can finish seconds or minutes later Adds latency; alert on queue depth and age Message queues
Autoscaling Sustained growth outrunning fixed capacity Load changes over minutes or hours Too slow for second-scale spikes; new instances need warm-up Scalability
Two meanings of "rate limiting"

Azure's pattern catalogue uses "Rate Limiting" for a client pacing its own calls to stay under a dependency's quota, and "Throttling" for the server enforcing a limit. Interviews usually mean the server side. Say which you mean.

04A step fails halfway

Anything with more than one write can stop in the middle. These patterns make partial work safe to repeat or to undo.

Pattern Failure it contains Use when Cost / gotcha Our page
Idempotency key A retry doing the work twice, such as a double charge Any write that a client or queue might resend Store keys with a TTL; reject the same key with a different body Idempotency
Saga with compensating steps Step 3 of 4 fails and steps 1 and 2 are already applied Flows across services: order, pay, ship Compensation isn't a rollback; others may have seen the in-between state Multi-step workflows
Transactional outbox The database commits but the event is never published You update a row and must also emit an event A relay to run and monitor; consumers see duplicates Outbox pattern
Competing consumers One worker crash stalling all background work Jobs whose messages are independent Order is lost across consumers; handle redelivery Message queues
Dead letter queue One bad message retried forever, blocking the rest Any queue consumer Someone has to own and drain the DLQ Dead letter queue

05One failure takes everything down

These patterns limit the blast radius: how many users one fault can reach.

Pattern Failure it contains Use when Cost / gotcha Our page
Redundancy (N+1 across zones) One machine or zone dying Every tier, from day one Copies that share power, config or a dependency fail together Single point of failure
Health endpoint monitoring Traffic still going to a dead instance Behind any load balancer or orchestrator Shallow checks lie; deep checks can mark every instance dead at once Heartbeats
Failover The primary dying and taking writes with it Stateful parts with a single writer Detection delay, a data-loss window, split brain Failover
Leader election Two nodes both doing the "only one of us" job Schedulers, primaries, singleton workers Needs fencing, because an old leader can wake up from a long pause Consensus algorithms
Cells (deployment stamps) A bad deploy or a noisy tenant hitting every user Large multi-tenant systems More copies to run; features that span cells get harder Fault tolerance
Backends for frontends A mobile-only spike or bug breaking web too Clients with very different needs More services to own; shared logic can drift apart API gateway

06We lost data or a region

Replication protects you from hardware. It does not protect you from yourself: a bad delete replicates just as faithfully as a good write.

Pattern Failure it contains Use when Cost / gotcha Our page
Replication A disk or node dying with the only copy Any data you can't recreate Async loses recent writes on failover; sync adds write latency Data replication
Backups with point-in-time restore A bad delete or corruption that replication copied everywhere Every database A backup you've never restored is a hope, not a plan Disaster recovery
Disaster recovery (stated RPO and RTO) A whole region gone The business can't wait for the region to come back Cost climbs steeply as RPO and RTO approach zero Disaster recovery
Checksums Silent corruption on disk or on the wire Anything stored or sent that must come back exact CPU per byte; you still need a good copy to repair from Checksums

07When patterns fight each other

Some combinations hurt. A senior candidate mentions these unprompted:

  • Retries at every layer. The Google SRE book's example: if three layers each make up to 4 attempts, one user click can become 4 × 4 × 4 = 64 attempts on an overloaded database. Retry at one layer, and give each process a retry budget.
  • Timeouts that don't nest. If the caller gives up at 1 second but your inner call waits 2, you're working for nobody. Pass the deadline down.
  • Automatic failover on async replication. The last few seconds of writes can vanish: fine for a feed, not a ledger.
  • Fallbacks that hide wrong answers. A cached product page is fine. A cached account balance is a correctness bug.
Many candidates recite patterns instead of naming failures

"I'd add circuit breakers, retries, bulkheads, rate limiting and a cache" sounds thorough and says nothing. Which dependency trips the breaker? What does the user see when it's open? Is the retried call idempotent? One pattern tied to a named failure and cost beats five names.

08Pick three

The drill to practise: name the three failures that would hurt most, then one pattern for each.

Try it yourself first

Before reading on, pick three patterns for each prompt and say which failure each one contains.

  1. A payment API that charges cards through an external provider.
  2. A news feed that ranks posts from people you follow.
  3. Ride matching that pairs riders with nearby drivers.

Here's what I'd pick:

Prompt Pattern The failure it contains
Payment API Idempotency key on every charge A client retry or timeout causing a double charge
Timeout plus circuit breaker on the provider, with "payment pending" as the fallback A slow provider tying up every checkout thread
Transactional outbox for "payment succeeded" events Money taken but the order never told
News feed Fallback to a cached or unranked feed The ranking service being down blanks the home screen
Queue-based load leveling for fan-out One celebrity post creating millions of writes at once
Load shedding of low-priority work (prefetch, analytics) first A traffic spike slowing the feed for everyone
Ride matching Cells by city or region One city's bad deploy or overload spreading everywhere
Heartbeats with timeouts for driver presence Offering rides to drivers whose app died minutes ago
Idempotent "accept ride" Two taps or a retry assigning one driver twice

Notice that the payment picks are about correctness, the feed picks about degrading gracefully, and the ride picks about freshness and blast radius. The product decides which failures matter, and that decides the patterns.

Go deeper: longer write-ups of nearly every pattern here are in Distributed systems patterns, and how these failures unfold in production is in Backend failure modes.

09How to say it in the interview

When I ask "what happens if this fails?", I'm listening for a named failure, a pattern that contains it, and its cost. Sweep the request path quickly, then let the interviewer pick where to go deeper.

In the interview

"Let me walk the request path. At the client, retries are safe because every write carries an idempotency key, and they back off with jitter. At the gateway, per-user rate limits stop one caller from starving others, and under real overload we shed background traffic first. Every call to the payment provider has a timeout below our own deadline and a circuit breaker, so when the provider is down we mark the payment pending instead of hanging. Order, payment and shipping run as a saga with compensating steps, and events go out through an outbox. The database has a synchronous standby in another zone plus point-in-time backups, for a zone loss and a bad delete. And I retry at one layer only, so an outage doesn't multiply load."

On the whiteboard

  • name the failure → pick the pattern → say the cost
  • slow / down dep: timeout, retry + jitter, breaker, fallback, bulkhead
  • overload: rate limit, shed load, queue, autoscale
  • half-done work: idempotency key, saga, outbox, DLQ
  • blast radius: N+1 zones, health checks, failover, cells
  • data / region: replicas + backups, RPO / RTO
  • retry at ONE layer (4 tries at each of 3 layers = 64)
Further reading (4)