Engineering Atlasby Abhishek Goyal
Course contents

Reliability: SLIs, SLOs and error budgets

Reliability for system design interviews: faults vs failures, reliability vs availability vs durability, SLIs, SLOs and SLAs, error budgets with burn-rate alerts, and why cutting MTTR beats raising MTBF.

FreeMust know for interviewsLesson12 min readBeginner → SeniorReviewed Oct 2026
On this page
  1. Picture it first: faults happen, failures are optional
  2. Reliability vs availability vs durability
  3. Measuring it: SLI, SLO and SLA
  4. Error budgets: turning a target into decisions
  5. MTBF and MTTR: the two levers
  6. The reliability toolbox, in one table
  7. Common traps
  8. How it shows up in design problems
  9. Map of this topic
  10. How to say it in the interview

🛡️ What is reliability?

A system is reliable when it keeps doing the right thing, for the right users, over time, even while parts of it break. Think of a car you trust: it starts every morning, it also gets you to work, and a flat tyre doesn't strand you because there's a spare in the boot. Disks die and deploys go wrong; reliability is making sure users don't notice.

For this lesson, we'll pin down the vocabulary the rest of the topic leans on: faults vs failures, reliability vs availability vs durability, SLIs, SLOs, SLAs, error budgets, and the two levers MTBF and MTTR. Interviewers love "how would you make this reliable?" because weak answers say "add more servers" and strong ones say "here's what I measure, here's the target, here's how fast I notice and recover".

Below the line (out of scope)

The 30-second answer

Reliability means the system keeps working correctly despite faults. A fault is one part misbehaving; a failure is when the user sees it, and the goal is to stop the first becoming the second. I measure it with SLIs, such as the share of requests that succeed fast enough, and I set an SLO, an internal target like 99.9% over 30 days. The SLA is the looser promise to customers, with penalties. The gap between 100% and the SLO is the error budget: while there's budget left we ship, when it's gone we slow down. Reliability has two levers, fail less often (MTBF) or recover faster (MTTR), and faster recovery is usually the cheaper win.

What you’ll learn

  • Separate a fault from a failure, and say where each technique breaks the chain
  • Tell reliability, availability and durability apart with one example each
  • Define an SLI, SLO and SLA for a real user journey, using percentiles instead of averages
  • Turn an SLO into an error budget in requests and minutes, and alert on burn rate
  • Use MTBF / (MTBF + MTTR) to show why recovering faster usually beats failing less often

01Picture it first: faults happen, failures are optional

Here's the most useful sentence in this topic: you can't prevent faults, but you can stop them becoming failures. A fault is one component deviating from what it should do: a disk dies, a deploy ships a bug, someone fat-fingers a config value. An error is the wrong state that results inside the system, like a timeout or a bad value in memory. A failure is when the system as a whole stops giving the user what they need: a 500, a wrong account balance, a page that takes 8 seconds.

Fig. 1

From fault to failure, and where you can break the chain

From fault to failure, and where you can break the chainThree boxes in a row. A fault, such as a dead disk, a bad deploy or a config typo, becomes an error inside the system, such as a timeout or a wrong value, which becomes a failure the user sees, such as a 500 or a wrong balance. Under the first arrow, redundancy and replication stop a fault from becoming an error. Under the second arrow, timeouts, retries and fallbacks stop an error from reaching the user. After the failure, fast detection and rollback or failover shorten how long it lasts.Where you break the chainFaulta component misbehavesdisk dies, bad deploy,config typo1Errorwrong state insidetimeout, bad value,exhausted pool2Failurethe user notices500, wrong balance,8 s page load3Contain the faultredundancy, replication,checksumsContain the errortimeouts, retries,fallbacks, circuit breakersShorten itdetect fast,roll back, fail overMTBF goes upwhen faults stayinvisible
Every technique sits on one of the arrows. Redundancy stops a fault becoming an error, timeouts and fallbacks stop an error reaching the user, and fast detection plus rollback shortens the failures that still get through.

Martin Kleppmann draws the same line in chapter 1 of Designing Data-Intensive Applications. It turns "how do I make sure nothing breaks?" (impossible) into "where in this chain do I catch it?" (designable). Faults come from three places:

Fault source Example Typical defence
Hardware A disk fails, a machine reboots, a zone loses power Redundancy across machines and zones
Software A bug that crashes every instance on the same bad input, a leak, a slow dependency Timeouts, gradual rollouts, isolation
Human A bad config push, a mistaken delete, a migration run against prod Automation, canaries, fast rollback

Notice the software row. A second machine helps with hardware faults, but a bug hits every copy at once. That's why reliability is more than "run three of everything".

02Reliability vs availability vs durability

Term Question it answers Measured as Example of losing it
Reliability Does it keep doing the right thing over time? Share of good requests, correctness, time between failures The API is up but charges some customers twice
Availability Can users reach it when they try? Uptime ratio or successful-request ratio The API returns 503 for 20 minutes
Durability Once stored, is the data still there later? Probability of not losing an object over a year A bucket loses files after a disk failure

A system can be available but not reliable: it answers every request, quickly, with the wrong number. It can also be durable but not available: your data is safe on disk while the service in front of it is down. Amazon S3 makes the split concrete: S3 Standard is designed for 99.999999999% (eleven nines) durability but 99.99% availability of objects over a year. Different promises, different numbers.

03Measuring it: SLI, SLO and SLA

"Make it reliable" isn't designable. "99.9% of checkout requests succeed in under 300 ms" is. Three acronyms turn the first into the second.

  • SLI (service level indicator): a number you measure. Usually a ratio: good events divided by all events. Success rate, latency under a threshold, freshness of data, durability.
  • SLO (service level objective): the target for that SLI over a window. "99.9% of requests good over a rolling 30 days." It's internal: your team owns it and acts on it.
  • SLA (service level agreement): a contract with customers that says what happens, usually money back as service credits, if you miss. It's a business document, not an engineering one.
Fig. 2

SLI, SLO, SLA and the error budget

SLI, SLO, SLA and the error budgetA ladder of three steps for a checkout API. The SLI is the measurement: the share of checkout requests that succeed in under 300 ms. The SLO is the internal target: 99.9 percent over 30 days. The SLA is the external contract: 99.5 percent, with service credits if missed. The gap between 100 percent and the SLO is the error budget: 0.1 percent, or about 43 minutes of full outage in 30 days. The SLA sits below the SLO so the team is warned before money is at stake.the gap you may spendSLI: what you measuregood checkout requests / all(good = 2xx in under 300 ms)1SLO: what you aim for (internal)99.9% of requests goodover a rolling 30 days2SLA: what you promise (contract)99.5% per month,or customers get credits3Error budget100% - 99.9% = 0.1%~43 min of fulloutage per 30 daysSLA looser than SLO: you get warned before you owe moneyBudget left? Ship faster. Budget gone? Freeze risky changes.
The SLA sits below the SLO on purpose. If your internal target is stricter than your contract, alerts fire while there's still room, long before anyone owes credits.

Three things make an SLI good. It measures what the user feels: a health check can be green while every checkout fails. It uses percentiles, not averages: an average of 120 ms can hide a p99 of 3 seconds. And it combines speed and success: "a 2xx in under 300 ms" counts slow successes as failures, which is how users experience them. Pick one or two SLIs per user journey (browse, add to cart, pay), not twenty per microservice.

04Error budgets: turning a target into decisions

The error budget is 100% minus the SLO: with a 99.9% SLO you may have 0.1% bad requests. It sounds like admitting defeat, but it's what stops reliability and feature work from fighting. While budget remains, the team ships and takes some risk. When it's spent, risky launches pause until the window recovers. Nobody argues about "reliable enough": the number decides.

Try it yourself first

Before reading on, work these out. Your checkout API serves 50 million requests in a 30-day window and has a 99.9% SLO. How many bad requests can you afford? And if the whole service went down, how many minutes of full outage would use the entire budget?

  • Budget in requests: 50,000,000 × 0.001 = 50,000 bad requests in 30 days.
  • Budget in time: 30 days is 43,200 minutes, and 43,200 × 0.001 = 43.2 minutes of total outage.

So one bad 40-minute deploy can burn nearly the whole month. That's why teams alert on burn rate, how fast the budget is being spent, rather than on single errors. Google's SRE Workbook suggests paging if 2% of a 30-day budget goes in one hour (a burn rate of 14.4, meaning 14.4 times faster than "exactly on budget"), and opening a ticket for a slow leak such as 10% over three days.

Google took a service down on purpose because it was too reliable

Google's SRE book describes the global instance of Chubby, its internal lock service. Real outages were so rare that other teams added dependencies on it as if it could never go down, and their services broke when it did. Google's answer: keep Chubby at its SLO but not far above it. In a quarter where no real failure has pulled availability down to the target, SRE takes the service down on purpose, which flushes out those hidden dependencies early. The lesson: people build on the reliability you actually deliver, not the one you wrote down, so overshooting an SLO by a lot isn't free.

05MTBF and MTTR: the two levers

MTBF (mean time between failures) is how long you go between incidents. MTTR (mean time to recovery) is how long each one lasts: time to detect, decide, and fix or roll back. For a single component, the fraction of time it's working is:

uptime fraction = MTBF / (MTBF + MTTR)

Now compare two investments for a service that fails about once every 30 days (720 hours) and takes an hour to recover:

Option MTBF MTTR Uptime, roughly
Today 720 h 60 min 99.861%
Fail half as often 1,440 h 60 min 99.931%
Recover 10× faster 720 h 6 min 99.986%

Doubling the time between failures, which means hunting down many rare causes, buys less than cutting recovery from an hour to six minutes. And recovery speed is engineering you control: SLI alerts, one-click rollback, automated failover. AWS's reliability design principles start in the same place: "automatically recover from failure" and "test recovery procedures".

What eats MTTR

In most incidents, the fix is fast and the noticing is slow. If your alert fires 15 minutes in and rollback takes 2, detection is 88% of the outage. Alert on user-facing SLIs, keep deploys small so rollback is safe, and make rollback the first move, not the last.

06The reliability toolbox, in one table

One line per pattern, keyed by the failure it contains. Each has its own page:

Pattern The failure it contains Our page
Timeouts, retries with backoff, bulkheads A slow or flaky dependency dragging everything down Fault tolerance
Circuit breaker Hammering a dependency that's already down Circuit breaker
Cache-aside Reads failing when the database has a bad minute Caching strategies
Competing consumers on a queue One worker crashing stalls all background work Message queues
Compensating transactions (sagas) A multi-step operation failing halfway Multi-step workflows
Event sourcing Losing the history of how state got where it is Event-driven architecture
Checksums Silent corruption on disk or on the wire Checksums

The full set, with "use when" and costs, is in the Reliability patterns cheat sheet. Azure's Well-Architected Framework keeps a broader catalogue, linked in the sources.

07Common traps

Many candidates answer "how do you make it reliable?" with only redundancy

"Three replicas behind a load balancer" handles one fault: a machine dying. It does nothing for a bad deploy that ships to all three, or a response that looks fine but is wrong. A senior answer names the SLI and SLO first, then detection (burn-rate alerts), containment (timeouts, isolation, gradual rollout) and recovery (rollback, failover), with redundancy as one item in that list.

Three more I hear often:

  • Uptime only. A payment system that's up but double-charges isn't reliable. Put correctness and durability in your SLIs when the product needs them.
  • SLOs nobody acts on. If it doesn't drive alerts or release decisions, it's decoration. Say what happens when the budget runs out.
  • Measuring only from the inside. Server-side success rate misses DNS failures, expired certificates and CDN problems. Add an outside probe, and use distributed tracing (Observability: logs, metrics and traces) to find where a slow request spent its time.

08How it shows up in design problems

"What happens when X fails?" shows up in the last 10 minutes of almost every round. Three problems lean on this page hardest:

  • Design a payment system: correctness is the SLI that matters, because "available but wrong" is the worst outcome.
  • Design metrics, monitoring and alerting: where SLIs come from, why you can't average percentiles across hosts, and the burn-rate alerts from this page.

Go deeper on what actually breaks in Backend failure modes, and on how data survives crashes in Disk, I/O and durability.

09Map of this topic

This page is the vocabulary. The rest turns it into design moves:

Page What it gives you in the interview
Availability and the nines Turn a target into downtime, and do the math for chains and replicas
Single point of failure A method to sweep your diagram and find the box that takes everything down
Failover What actually happens when the primary dies, and how long it takes
Fault tolerance Timeouts, retries, bulkheads, degradation and load shedding, and when each applies
Reliability patterns cheat sheet Every pattern on one page, grouped by the failure it contains

10How to say it in the interview

When I ask "how would you make this reliable?", I'm listening for a measurable target, how you'd notice you're missing it, and how fast you'd recover. Candidates who start with the SLI sound like they've been on call.

In the interview

"Let me define reliable for this system first. For checkout, my SLI is the share of requests that return success in under 300 ms, measured at the load balancer and with an outside probe. My SLO is 99.9% over 30 days, which gives an error budget of about 43 minutes of full outage or 0.1% of requests. The customer SLA can be looser, say 99.5%, so we're warned well before we owe credits. I'd alert on burn rate: page if we burn 2% of the budget in an hour, ticket if it's a slow leak. To keep faults from becoming failures, I'd use redundancy across zones, timeouts and bounded retries on every call, and a circuit breaker on the payment provider. And because recovering fast beats failing rarely, deploys are small and canaried, rollback is one click, and the database fails over automatically. If the budget runs out, we pause risky launches until it recovers."

On the whiteboard

  • fault (part breaks) → error (bad state) → failure (user sees it)
  • reliability ≠ availability ≠ durability
  • SLI = good / total (success AND fast), use p99 not avg
  • SLO = internal target, e.g. 99.9% / 30 d
  • SLA = contract, looser than SLO, credits
  • error budget = 1 - SLO → 43.2 min / 30 d at 99.9%
  • alert on burn rate (2% in 1 h = 14.4×)
  • uptime = MTBF / (MTBF + MTTR) → cut MTTR first
Further reading (5)