Distributed Systems & Backend Patterns, End to End

Every pattern below exists to answer one question: what happens when the thing you're depending on doesn't work the way you assumed? A dependency gets slow. A network partitions. A cache goes cold at the worst possible moment. Ten thousand requests arrive in the same second. Each pattern is a specific, named answer to one of those moments — and the patterns compose: a real production system runs a dozen of these simultaneously, layered on top of each other.

This guide groups them the way they actually get reached for — by the failure or bottleneck each one is defending against — then closes with a cheat sheet mapping symptom to pattern.

§1Resilience: surviving a dependency that's failing

Resilience patterns share one premise: something you depend on will fail, and it will fail in the middle of your request. The goal isn't preventing that failure — you usually can't — it's making sure one failing dependency doesn't take your whole system down with it.

Exponential Backoff

Solves: retry storms — a fleet of clients all retrying a failed call at the same instant, which is often what tips a struggling service over into an outage.

The idea is simple: each retry waits longer than the last, growing exponentially, so a client doesn't hammer a struggling service at a fixed interval. delay = base * 2^attempt. Without jitter, every client computes the exact same delay sequence and they all retry in lockstep anyway — jitter (a random offset added to each delay) is what actually breaks the synchronization.

def backoff_delay(attempt, base=1.0, cap=30.0):
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)   # "full jitter" -- AWS's recommended formula

for attempt in range(max_retries):
    try:
        return call_service()
    except TransientError:
        time.sleep(backoff_delay(attempt))
raise MaxRetriesExceeded()

Circuit Breaker

Solves: cascading failures — a slow or dead downstream service causing every upstream caller to pile up threads waiting on it, until the caller itself runs out of resources and fails too.

Modeled directly on the electrical version: too many failures and the circuit "trips," and calls fail immediately without ever touching the struggling service — giving it room to recover instead of being hit by requests while it's already down.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=30):
        self.state = "CLOSED"
        self.failures = 0
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.opened_at = None

    def call(self, fn, *args):
        if self.state == "OPEN":
            if time.time() - self.opened_at > self.reset_timeout:
                self.state = "HALF_OPEN"
            else:
                raise CircuitOpenError()
        try:
            result = fn(*args)
            self._on_success()
            return result
        except Exception:
            self._on_failure()
            raise

    def _on_success(self):
        self.failures = 0
        self.state = "CLOSED"

    def _on_failure(self):
        self.failures += 1
        if self.state == "HALF_OPEN" or self.failures >= self.failure_threshold:
            self.state = "OPEN"
            self.opened_at = time.time()

Retry Pattern

Solves: transient failures — a blip, a dropped packet, a momentary overload — that would succeed if simply attempted again a moment later.

The pattern only makes sense for errors that are actually transient and for operations that are safe to repeat. Retrying a 500 from an idempotent GET is safe. Retrying a non-idempotent "charge this card" POST without a dedupe key can double-charge a customer — see the Idempotency pattern later in this guide before wiring retry onto anything that mutates state.

Timeout Pattern

Solves: hanging requests — a call with no timeout will wait forever for a dependency that will never respond, holding a thread and its resources hostage the entire time.

Every network call needs an explicit timeout — there is no safe default of "wait indefinitely." The harder part is picking the number: too short and you abort calls that would have succeeded; too long and a struggling dependency ties up your resources anyway, just more slowly.

Bulkhead Pattern

Solves: failure isolation — one overwhelmed dependency exhausting the shared thread pool or connection pool that every other, unrelated dependency also relies on.

Named after ship design: watertight compartments mean one flooded section doesn't sink the whole vessel. In software, that means giving each downstream dependency its own dedicated pool of threads or connections, sized to that dependency's own risk and importance — so a slow, low-priority dependency can never starve the pool that a critical one depends on.

Fail Fast

Solves: wasted resources spent on a request that's already known to be doomed — checking cheap, definitive failure conditions immediately rather than partway through expensive processing.

Validate inputs, check circuit breaker state, and check resource availability at the very top of a request handler, before acquiring any locks, opening any connections, or doing any real work. A request that's going to be rejected should be rejected in microseconds, not after it's already consumed a database connection.

Fallback Pattern

Solves: graceful degradation — when a dependency is unavailable, returning something imperfect but useful is almost always better for the user than returning nothing.

Hedged Requests

Solves: tail latency — the small fraction of requests that hit an unusually slow replica or path, which dominate p99 latency even when the median is fine.

Fire the same request to a second replica if the first hasn't responded within some threshold (often the p95 latency for that call), and take whichever answer comes back first, canceling the other. This trades a small amount of duplicate load for a large cut in tail latency — Google's "The Tail at Scale" paper found even a small hedge rate meaningfully improved p99s.

Dead Letter Queue (DLQ)

Solves: failed message handling — a message that a consumer can't process (bad data, a downstream bug, an exhausted retry budget) either blocks the entire queue behind it, or gets silently dropped and lost forever. Neither is acceptable.

After N failed processing attempts, move the message to a separate dead letter queue instead of retrying forever or discarding it. This unblocks the main queue immediately and preserves the message for inspection, alerting, and manual or automated reprocessing later.

§2Load management: surviving too many requests, not too few

Resilience patterns assume a dependency is failing. Load management patterns assume the opposite problem: everything is healthy, there's just more demand than capacity, and something has to give — either you shape the traffic, or the traffic shapes your outage.

Rate Limiting

Solves: too many requests from a single client or in aggregate overwhelming a service, intentionally or not.

Rate limiting rejects requests once a client crosses a defined threshold, protecting the service from a single noisy client and giving fair access across many. The interesting engineering is entirely in which algorithm — see Token Bucket, Leaky Bucket, and Sliding Window below, each with real trade-offs.

Token Bucket

Solves: smoothing bursty traffic while still allowing legitimate short bursts, rather than enforcing a rigid constant rate that would reject perfectly reasonable spikes.

A bucket holds up to N tokens. Tokens refill at a fixed rate. Every request consumes one token; if the bucket is empty, the request is rejected or queued. Because tokens can accumulate while idle, a client that hasn't made a request in a while can burst up to the bucket's full capacity — this is the key difference from Leaky Bucket.

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate  # tokens per second
        self.last_refill = time.time()

    def allow(self):
        now = time.time()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Leaky Bucket

Solves: the same problem as token bucket, but where you need a strictly constant output rate — no bursts at all — because the downstream system genuinely cannot absorb them.

Requests fill a queue (the "bucket"); the queue drains at a fixed rate regardless of how fast it fills. Unlike token bucket, there's no accumulated allowance for bursting — the bucket smooths everything to one constant outflow rate, so it's the right choice when the consumer's constraint is truly about rate, not burst tolerance.

Sliding Window (and Fixed Window)

Solves: accurate rate limiting at window boundaries — fixed windows let a client send double the intended rate by timing requests around the boundary.

Queue-Based Load Leveling

Solves: sudden traffic spikes that would overwhelm a downstream service if applied directly — decoupling producers from consumers with a buffer in between.

Instead of calling a downstream service directly, requests are placed on a queue; the downstream service consumes from that queue at whatever pace it can sustain. The spike is absorbed by the queue's depth, and the consumer never sees more load than it can actually handle — at the cost of added latency for whoever is waiting on the result.

Backpressure

Solves: slow consumers being overwhelmed by fast producers — without backpressure, a fast producer either OOMs an unbounded queue or silently drops data.

The consumer signals upstream — "slow down" — and the producer respects that signal rather than pushing regardless. This can be as simple as a bounded queue that blocks the producer when full, or as explicit as TCP's own flow control window, or a reactive streams protocol like Flow.Subscription.request(n) where the consumer explicitly pulls only as much as it can process.

Load Shedding

Solves: a system already over capacity — rather than trying to serve everything slowly and risk failing all of it, load shedding deliberately drops low-priority requests to protect the ones that matter and keep the system itself alive.

Priority Queue (for traffic)

Solves: "VIP" traffic — some requests (a paying customer, a critical internal service, a health check) need to jump ahead of routine background traffic when a system is under load.

Tag requests with a priority at ingress and drain higher-priority queues first. This is what makes load shedding selective rather than random — shedding decisions become "drop from the lowest-priority queue first," not "drop whatever happens to be oldest."

§3Caching: trading correctness-freshness for speed

Every caching pattern is the same trade dressed differently: some data is expensive to compute or fetch, so you keep a copy somewhere faster — and now you own the question of when that copy stops matching the truth.

Cache Aside (Lazy Loading)

Solves: read-heavy systems where most data is never touched — cache aside only populates the cache with data that's actually been requested, rather than pre-loading everything.

def get_user(user_id):
    cached = cache.get(f"user:{user_id}")
    if cached:
        return cached
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)
    cache.set(f"user:{user_id}", user, ttl=300)
    return user

Read Through / Write Through

Solves: the same miss-then-fetch problem as cache aside, but pushes that logic into the caching layer itself rather than the application — the app only ever talks to the cache, and the cache transparently handles the database on a miss (read through) or writes to both synchronously (write through).

Write Behind (Write Back)

Solves: write latency — write through pays the full database write cost on every request; write behind acknowledges the write once it's in the cache and flushes to the database asynchronously afterward.

Refresh Ahead is the read-side complement: proactively refresh a cache entry before it expires (typically when it's accessed and its TTL is getting close to zero), so a legitimately popular key never actually experiences a cache miss.

Cache Stampede Prevention

Solves: a cache miss storm — a popular key expires, and hundreds of concurrent requests all miss simultaneously and hammer the database at once with the identical query, sometimes hard enough to take the database down.

Bloom Filter

Solves: cache penetration — repeated lookups for keys that don't exist at all still miss the cache and hit the database every single time, since there's nothing to cache for a key with no data.

A Bloom filter is a compact probabilistic structure that answers "might this key exist?" with no false negatives but a tunable false-positive rate. Check it before querying the database: a definite "no" skips the database entirely; a "maybe" proceeds as normal. It costs a small amount of memory to eliminate a large class of pointless database hits from bots, scrapers, or bad IDs.

Hot Key Replication

Solves: hot partitions — a single extremely popular key (a viral post, a celebrity's profile) can overload the one cache node or shard responsible for it, even while every other node sits idle.

Replicate the hot key across multiple cache nodes and route reads to a randomly chosen replica, spreading load for that one key across several machines instead of concentrating it on one. Detecting which keys are "hot" in the first place usually requires sampling access frequency in real time.

§4Messaging: decoupling producers from consumers in time

Synchronous request/response ties a caller's fate to the callee's availability and speed. Messaging patterns break that coupling — the producer emits and moves on, the consumer processes on its own schedule, and the hard problems that surface are ordering, duplication, and partial failure across that gap.

Publish/Subscribe

Solves: event broadcasting — one event often needs to trigger action in several unrelated downstream systems (billing, analytics, notifications), and the publisher shouldn't need to know who's listening.

A publisher emits to a topic; any number of subscribers receive a copy independently. Adding a new subscriber never requires changing the publisher — this decoupling is the entire point, and it's what lets a system's downstream consumers grow over time without touching upstream code.

Event Sourcing

Solves: audit history — storing only the current state of an entity throws away exactly how it got there, which is often exactly what you need for debugging, compliance, or rebuilding derived views.

Instead of storing current state, store the full sequence of events that produced it (AccountOpened, Deposited $50, Withdrew $20) and derive current state by replaying them. The event log becomes the single source of truth; any view of "current state" is just a projection that can be rebuilt from scratch if it's ever wrong.

CQRS (Command Query Responsibility Segregation)

Solves: read and write workloads with very different shapes — a single model optimized for one is usually a poor fit for the other, especially at scale.

Split the write model (commands, normalized, optimized for correctness and validation) from the read model (queries, often denormalized, optimized for the exact shape a UI needs). The two can even live in entirely separate datastores, kept in sync via the events emitted by the write side — which is why CQRS and Event Sourcing so often show up paired together.

Saga Pattern

Solves: distributed transactions — a single business operation (place an order) that spans multiple services (payment, inventory, shipping) has no single database transaction that can wrap all of it atomically.

A saga breaks the operation into a sequence of local transactions, one per service, each with a defined compensating transaction to undo it if a later step fails. If step 3 of 4 fails, the saga runs the compensations for steps 2 and 1 in reverse — refund the payment, release the reserved inventory — rather than leaving the system in a half-completed state.

Outbox Pattern

Solves: reliable event publishing — writing to your database and separately publishing an event to a message broker is two operations with no shared transaction; a crash between them either loses the event or publishes one for a database write that never actually committed.

Write the event to an "outbox" table in the same database transaction as the actual business data change — so both succeed or both roll back together, atomically. A separate process then reads the outbox table and publishes each event to the broker, marking it published once confirmed.

The Inbox Pattern is the consumer-side complement: track which message IDs have already been processed in a dedicated table, so a redelivered message (which brokers will do — see Delivery Guarantees) is recognized and skipped rather than processed twice.

Idempotent Consumer

Solves: duplicate messages — most real message brokers guarantee at-least-once delivery, meaning a consumer must expect to see the same message more than once and handle it safely.

Design message handlers so processing the same message twice produces the same result as processing it once — track processed message IDs (an inbox table, a Redis set with a TTL) and skip anything already seen. See the Idempotency pattern in the Reliability section for the general principle this pattern applies specifically to consumers.

Competing Consumers

Solves: parallel processing — a single consumer processing a queue serially caps throughput at that one consumer's speed, even when the work is fully parallelizable.

Run multiple consumer instances against the same queue; the broker hands each message to exactly one of them, so throughput scales roughly linearly with the number of consumers. The catch: message ordering across the whole queue is no longer guaranteed once more than one consumer is pulling from it — if order matters for a given entity, route related messages to the same consumer (partitioning by key, as Kafka does) rather than trying to preserve global order.

§5Database patterns: scaling storage without losing your data

A single database server can only hold so much data and answer so many queries per second. These patterns are how systems keep growing past that ceiling — splitting data up, copying it around, and controlling who gets to change it and when.

Sharding

Solves: horizontal scaling of storage and write throughput — a dataset too large or too write-heavy for one machine gets split across many, each holding a subset ("shard") of the total data.

The critical decision is the shard key — the field used to decide which shard a given row lives on. A well-chosen key spreads load evenly; a poor one (say, sharding by signup date when all recent traffic hits recent signups) creates a hot shard that defeats the entire point of sharding.

Read Replicas & Leader-Follower Replication

Solves: read scalability and high availability — most systems read far more than they write, so directing all reads at one primary wastes the ability to scale reads independently.

One leader accepts writes and replicates them (synchronously or asynchronously) to one or more followers, which serve read traffic. Asynchronous replication means followers can lag behind the leader by some amount — this is replication lag, and it's the direct cause of "I just wrote something and then couldn't see it on the very next read."

Leader-leader (multi-leader) replication accepts writes at more than one node, usually one per region for low-latency local writes — at the cost of needing an explicit conflict-resolution strategy (last-write-wins, CRDTs, or vector clocks — see below) for when two leaders accept conflicting writes to the same record.

Consistent Hashing

Solves: even distribution across nodes while minimizing data movement when a node is added or removed — naive hash(key) % N reshuffles almost every key's assigned node the instant N changes.

Nodes and keys are both hashed onto the same conceptual ring; a key belongs to the first node found walking clockwise from its position. Adding or removing one node only reassigns the keys between it and its ring neighbor — every other key stays exactly where it was.

Materialized Views

Solves: expensive queries run repeatedly — a complex join or aggregation recomputed on every request wastes work when the underlying data changes far less often than it's read.

Precompute and store the query's result as its own table, refreshed periodically or on a trigger from the underlying data changing. Reads become a simple lookup; the cost moves to keeping the view updated, which is a deliberate trade of freshness for read speed.

Optimistic vs. Pessimistic Locking

Solves: concurrent updates to the same row — two transactions modifying the same data at once need a strategy to prevent one from silently overwriting the other's work.

§6Distributed systems: agreement without a shared clock or a single point of truth

Everything in this section exists because of one hard fact: in a distributed system, there is no shared, reliable notion of "now," and any message can be delayed, lost, or arrive out of order. You cannot tell the difference between "that node is slow" and "that node is dead" from the outside — every pattern here is a strategy for making progress anyway, despite that fundamental uncertainty.

Leader Election

Solves: needing exactly one coordinator among a group of equivalent nodes — for accepting writes, scheduling work, or breaking ties — without a human manually assigning the role and without two nodes both believing they're in charge at once.

Nodes run a protocol (Bully algorithm, Raft's leader election, or a lease held in a coordination service like ZooKeeper or etcd) to agree on a single leader. The leader typically holds a time-bound lease it must periodically renew; if it stops renewing (because it crashed, or is partitioned away), the lease expires and the remaining nodes elect a new one.

Service Discovery

Solves: dynamic instances — in an environment where service instances are created and destroyed constantly (autoscaling, deploys, crashes), hardcoded IP addresses break immediately.

Either way, instances register themselves on startup and deregister (or get removed via failed health checks) on shutdown — the registry's accuracy depends entirely on this registration/deregistration discipline actually being reliable.

Heartbeat

Solves: failure detection — in a distributed system, you can't directly observe whether a remote node is alive; a heartbeat is the mechanism that lets you infer it.

Each node periodically sends a lightweight "I'm alive" signal. Miss enough consecutive heartbeats and the node is presumed dead — but this is a probabilistic judgment, not a certainty: a node can be alive but partitioned from the network, indistinguishable from actually being dead to everyone else. This ambiguity is precisely why distributed systems need consensus protocols rather than simple heartbeat-based assumptions for anything safety-critical.

Gossip Protocol

Solves: cluster-wide communication without every node needing to talk to every other node directly — a naive all-to-all broadcast doesn't scale past a small cluster size.

Each node periodically shares what it knows with a small, random subset of other nodes; information spreads exponentially through the cluster the way a rumor spreads through a social network, reaching every node in O(log N) rounds without any single node needing global connectivity. Cassandra and Consul both use gossip for cluster membership and failure detection.

Distributed Lock

Solves: shared resource access across multiple processes on different machines — a normal in-process mutex only protects against concurrent access within one process; a distributed lock is needed when the contenders are separate machines entirely.

Typically implemented on top of a coordination service (ZooKeeper, etcd, or Redis via the Redlock algorithm) as a lease with a TTL: a client acquires the lock, does its work, and releases it, or the lease expires automatically if the client crashes mid-operation.

Two-Phase Commit (2PC) & Three-Phase Commit (3PC)

Solves: strong, atomic consistency across multiple independent databases or services participating in one logical transaction.

Three-Phase Commit adds an intermediate "pre-commit" acknowledgment step specifically to reduce (not eliminate) this blocking window, at the cost of one more network round trip on every transaction. In practice, most modern systems avoid both altogether in favor of the Saga pattern's eventual consistency, precisely because blocking on a coordinator is such an availability risk.

Quorum Reads/Writes

Solves: tunable consistency in a replicated system — rather than requiring every replica to acknowledge a write (which kills availability the moment any one replica is slow) or just one (which risks reading stale data), quorum lets you require a majority.

With N replicas, a write requires acknowledgment from W of them, and a read requires responses from R of them. The guarantee W + R > N ensures every read overlaps with the most recent write in at least one replica — giving you strong consistency without needing all N nodes to participate in every operation.

Consensus (Raft / Paxos)

Solves: getting a cluster of unreliable nodes, communicating over an unreliable network, to agree on a single value or sequence of operations — the foundational problem underneath leader election, distributed locks, and replicated logs alike.

Raft (designed explicitly to be more understandable than Paxos) works in rounds called terms: a leader is elected per term and replicates a log of operations to followers, committing an entry only once a majority of nodes have durably stored it. Paxos solves the identical underlying problem with a more general but notoriously harder-to-implement-correctly two-phase proposal/accept protocol. Both guarantee safety (no two nodes disagree on a committed value) as long as a majority of nodes are healthy and can communicate.

Vector Clocks

Solves: conflict resolution when there's no global clock to say definitively which of two concurrent writes "happened first" — wall-clock timestamps from different machines can't be trusted to agree, and even if they could, two writes can be genuinely concurrent with no real ordering between them.

Each node keeps a vector of counters, one per node, incremented on every local update and merged (taking the max per position) on every message received. Comparing two vector clocks tells you whether one write causally preceded the other, or whether they were truly concurrent and need explicit conflict resolution (last-write-wins, application-level merge, or presenting both to the user, as early Amazon Dynamo famously did with shopping carts).

§7Traffic routing: getting requests to the right place, safely

Once you have more than one instance of a service — or more than one version of it — something has to decide which instance handles which request, and how a new version gets introduced without breaking everything currently in flight.

Load Balancer

Solves: traffic distribution — spreading incoming requests across multiple backend instances so no single instance is overwhelmed while others sit idle.

Sticky Sessions pin a given client to the same backend instance for the life of their session — necessary when session state lives in that instance's memory rather than a shared store, at the cost of uneven load if some clients are far more active than others.

API Gateway

Solves: centralized routing and cross-cutting concerns — without a gateway, every client needs to know the address of every backend service, and concerns like auth, rate limiting, and logging get duplicated into each service individually.

A single entry point that routes requests to the correct backend service and handles authentication, rate limiting, request/response transformation, and logging in one place. Related: a Reverse Proxy does the routing/forwarding piece without necessarily owning the cross-cutting policy layer — an API gateway is a reverse proxy with opinions.

A BFF (Backend for Frontend) takes this further with a dedicated gateway per client type (web, iOS, Android), each shaping responses exactly to that client's needs rather than forcing one generic API to serve all of them equally awkwardly.

Service Mesh

Solves: service-to-service communication concerns (retries, timeouts, mTLS, observability, traffic splitting) at scale, without embedding that logic in every service's own codebase in every language they happen to be written in.

A sidecar proxy (Envoy is the most common) runs alongside every service instance and intercepts all its network traffic, applying policy uniformly regardless of what language the service itself is written in. Istio and Linkerd are control planes that configure the sidecars fleet-wide.

Blue-Green Deployment

Solves: zero-downtime releases — deploying a new version by replacing the old one in place risks a window where neither version is fully healthy.

Run two identical environments, "blue" (current production) and "green" (new version). Deploy and fully test the new version on green while blue keeps serving all live traffic, then switch the router to green atomically. If anything's wrong, switch back to blue instantly — the previous version is still running, not torn down.

Canary Deployment

Solves: safe rollout — blue-green switches 100% of traffic at once; canary reduces the blast radius of a bad release by exposing it to a small slice of real traffic first.

Route a small percentage of traffic (often 1–5%) to the new version, watch error rates and latency closely, and ramp up gradually if metrics stay healthy — or roll back instantly, affecting only that small slice, if they don't. Shadow Traffic goes a step further for the most cautious rollouts: mirror real production traffic to the new version without it affecting any real user, comparing its outputs against the current version silently.

Feature Flags

Solves: controlled releases decoupled from deployment — without flags, "ship the code" and "turn on the feature" are the same event, which means the only way to disable a bad feature is another deploy.

Wrap new functionality in a runtime-checked flag so it can be toggled, gradually rolled out to a percentage of users, or targeted at specific segments — all without a new deploy. This also enables A/B testing and instant kill-switches for anything that turns out to be broken in production.

§8Reliability: doing exactly the right amount of work, exactly once (or knowing when you can't)

Networks drop acknowledgments even when the underlying operation succeeded. This section is about the guarantees you can actually make when you can't always tell whether something already happened.

Idempotency

Solves: safe retries — if retrying an operation is going to be a core part of how the system handles failure (and it always is), that operation needs to produce the same result no matter how many times it's actually executed.

The standard mechanism is an idempotency key: the client generates a unique key per logical operation and sends it with the request; the server checks whether it's already processed that key and, if so, returns the original result instead of executing the operation again.

def charge_card(idempotency_key, amount, card):
    existing = db.get_by_idempotency_key(idempotency_key)
    if existing:
        return existing.result   # already processed -- return the original outcome
    result = payment_processor.charge(amount, card)
    db.save_idempotency_record(idempotency_key, result)
    return result

Delivery Guarantees: At-Least-Once, At-Most-Once, Exactly-Once

Solves: defining precisely what a messaging system promises about how many times a message will be delivered — a prerequisite for knowing what your consumer actually needs to defend against.

Checkpointing & Snapshotting

Solves: recovery after a crash — without periodic checkpoints, a long-running stream processing job or stateful service has to replay its entire history from the beginning after any failure.

Periodically persist enough state to resume from that point rather than from the start — a stream processor saves its position in the input log plus any accumulated aggregation state; a database takes a full snapshot periodically so its write-ahead log doesn't have to be replayed from the beginning of time. Snapshotting is checkpointing applied to full system state rather than a single job's progress.

Health Checks & Watchdog

Solves: detecting failures automatically rather than relying on a human noticing — the prerequisite for automated recovery, load balancer routing decisions, and orchestrator restart decisions.

§9Scalability: adding capacity without adding coupling

Scaling is easy in theory — add more machines — and hard in practice, because most of the difficulty is in the assumptions your system quietly baked in about running on exactly one machine.

Horizontal vs. Vertical Scaling

Auto Scaling is horizontal scaling made dynamic: instance count adjusts automatically based on real-time load (CPU, queue depth, request rate) rather than a fixed, manually-set count — capacity grows to meet demand and shrinks back down when it passes, rather than being permanently provisioned for peak.

Stateless Services

Solves: the actual prerequisite for horizontal scaling and load balancing to work at all — if a service holds session or request state in its own memory, requests from the same client must always land on the same instance, defeating the point of having many interchangeable instances.

Push state out of the application process entirely — into a shared cache, a database, or a token the client holds itself (a signed JWT, for instance) — so that any instance can handle any request. This is also precisely what makes Blue-Green and Canary deployments, and auto scaling in general, straightforward: instances become disposable and interchangeable.

Worker Pools & Thread/Connection Pools

Solves: controlled concurrency — creating a new thread or a new database connection per task is expensive (see the thread pool internals covered in Java concurrency guides); pools reuse a fixed, warm set instead.

A worker pool applies the same idea at the process or job level: a fixed number of worker processes pull tasks from a shared queue, giving parallel execution with a hard, predictable ceiling on total resource usage — the same shape as a thread pool, just at a coarser grain. Microservices apply the same underlying idea at the architecture level: splitting a system so each piece can be scaled independently, based on its own specific load, rather than scaling one large monolith uniformly regardless of which part is actually the bottleneck.

§10Security: who is this, and what are they allowed to do

Every distributed system eventually needs to answer two separate questions for every request: who is making it (authentication), and what are they allowed to do (authorization) — plus, once you have more than one service, how services trust each other.

OAuth2, JWT, RBAC, ABAC, API Keys

mTLS & Secrets Management

mTLS (mutual TLS) solves service authentication in the reverse direction from normal TLS: not just the client verifying the server's certificate, but the server also verifying the client's — the foundation of "zero trust" service-to-service communication inside a service mesh, where no internal call is trusted just because it originated inside the network perimeter.

Secrets Management solves credential storage: database passwords, API keys, and certificates need to exist somewhere, and "hardcoded in a config file checked into git" is the single most common real-world security failure. A dedicated secrets manager (Vault, AWS Secrets Manager) stores them encrypted, controls access via policy, and can rotate them automatically.

§11Observability: understanding a system you cannot fully see

In a distributed system, no single machine has the whole picture of what happened for any given request. Observability patterns are how you reconstruct that picture after the fact.

Distributed Tracing & Correlation IDs

Solves: request tracking across service boundaries — a single user request in a microservices architecture might touch a dozen services, and understanding where time was actually spent (or where it failed) requires following that one request's path across all of them.

A correlation ID is generated at the edge and passed through every downstream call, tagging every log line and span with the same identifier so they can all be reassembled later. Distributed tracing (Jaeger, Zipkin) builds on this: each service creates a "span" recording how long its part took, and spans are linked into a full trace showing the request's entire path and timing, service by service.

Structured Logging & Metrics

Structured Logging solves searchable logs: plain-text log lines are only searchable by string matching; logging as structured key-value data (JSON) makes logs queryable by field — "show me every request with status=500 and service=payment" — rather than grepping for substrings.

Metrics solve system health at a glance: counters, gauges, and histograms (request rate, error rate, p50/p95/p99 latency) aggregated over time, cheap enough to collect at high volume and query in real time, unlike full log lines. OpenTelemetry has become the de facto standard for emitting all three — traces, logs, and metrics — through one unified, vendor-neutral instrumentation layer.

§12Advanced patterns: the vocabulary for everything else

These don't fit neatly into one failure category — they're structural patterns for organizing code and traffic that show up constantly once a system has any real size.

Cheat sheet: symptom → pattern

Resources