Backend Under Load: Eight Failure Modes

Interviewers at this level rarely want a fresh whiteboard architecture. They want to know what you do when the architecture you already drew starts to buckle — because in production, every one of these eight failure modes shows up sooner or later, usually more than one at a time. The skill being tested is symptom → correlated metric → root cause → fix, not trivia recall.

01Connection pool saturation

Your application doesn't talk to the database directly on every request. It borrows a connection from a fixed-size pool (HikariCP in Java is the canonical example), uses it, and returns it. When every connection is checked out and requests keep arriving, new requests queue behind the pool. If they wait long enough, they time out.

Why it happens

• Pool sized for average load, not actual peak concurrency.
• Connections held too long — long transactions, leaked connections that never return, N+1 queries inside a single transaction.
• A downstream slowdown (slow queries, a stalled dependency) means each connection is held longer than usual — the pool's effective throughput drops even though its size hasn't changed.
• Fleet math is the real trap: 50 pods × a pool size of 20 offers 1,000 simultaneous connections to a database that may handle a few hundred well. The pool protects a single instance — nothing protects the database as a whole unless you design for it.

Diagnosis

• Pool metrics: active connections ≈ max pool size, plus a growing "waiting for connection" queue.
• Symptom: SQLTransientConnectionException: Connection is not available, request timed out (HikariCP), or the equivalent driver timeout elsewhere.
• Correlate pool exhaustion with query latency, not just request volume — often the pool wasn't undersized, the queries got slow.

Fix / prevention

• Right-size the pool using HikariCP's starting-point formula, then tune with real load tests: connections ≈ (core_count × 2) + effective_spindle_count. Bigger isn't always better — excess connections add context-switching and contention on the DB itself.
• Fail fast: set an aggressive connectionTimeout (2–3s) so requests error out visibly instead of queuing silently.
• Cap total connections across the fleet, not per instance — think in terms of a database connection budget split across replicas, batch jobs, and admin tooling.
• Put a multiplexer/proxy (PgBouncer, ProxySQL, Oracle CMAN) between app and DB to decouple "connections the app thinks it opened" from "connections the DB actually maintains." PgBouncer's transaction pooling mode in particular can serve hundreds of app connections from a much smaller real pool.
• Turn on leak detection (HikariCP's leak-detection-threshold) to catch code paths that check out a connection and never return it.
• Watch for autoscaling "connection storms": 50 new pods opening a full pool at boot can spike DB CPU in seconds. Stagger warm-up — start with minimum-idle: 1 and grow lazily.

spring.datasource.hikari.maximum-pool-size=20
spring.datasource.hikari.minimum-idle=1
spring.datasource.hikari.connection-timeout=2500
spring.datasource.hikari.leak-detection-threshold=30000
spring.datasource.hikari.validation-timeout=1000

02Cache miss storm

A cache exists to shield the database from repeated reads of the same data. A cache miss storm happens when a large chunk of the cache becomes invalid or unavailable at once — a cache node restarts, a deploy flushes it, or many keys share a TTL and expire together — and a flood of requests that would normally hit the cache all fall through to the database simultaneously.

Why it happens

• Synchronized TTLs: many keys cached at the same moment with the same expiry all die together.
• Cold cache after deploy or restart: a new cache instance, or a cluster failover, starts empty.
• Eviction under memory pressure: LRU eviction removes hot keys exactly when you need them most.

Diagnosis

• Sudden, correlated spike in miss rate across many keys (contrast with Thundering Herd below, which is one key).
• DB read QPS spikes in lockstep with a cache hit-rate drop.
• Usually visible right after a deploy, a cache cluster resize, or a node failover.

Fix / prevention

• Jittered TTLs — add randomness to expiry: TTL = base ± random(0, jitter), so keys don't all die at the same instant.
• Cache warming before cutting traffic over to a new node or cluster, especially post-deploy.
• Graceful degradation: serve slightly stale data (stale-while-revalidate) instead of forcing every reader to the DB.
• Multi-tier caching: local in-process cache + shared Redis/Memcached, so a shared-cache blip doesn't send 100% of traffic straight to the DB.
• Read-through with request coalescing (see Thundering Herd) so even a full flush produces one DB query per key, not one per concurrent request.

03Thundering herd

An especially vicious, narrower case of a cache miss: one single hot key expires, and every one of its many concurrent readers independently notices the miss and hits the database at the same instant to regenerate it. The name borrows from the general OS-level phenomenon where many threads blocked on the same event all wake up simultaneously, but only one can actually proceed — the rest collide and go back to sleep, wasting the cycles it took to wake them.

A widely-used mental model: a product recommendation cache key expires at scale, and tens of thousands of concurrent requests hit the primary database within seconds — CPU spikes toward saturation, checkout pages slow to a crawl, and the root cause is "just" one expired cache key.

Diagnosis

• One specific key or query pattern dominates the miss spike (versus the broad, multi-key pattern of a cache miss storm).
• DB slow query log shows the exact same query running dozens or hundreds of times concurrently.
• Correlates precisely with the TTL of one hot object.

Fix / prevention

• Request coalescing / single-flighting: on a miss, the first request acquires a lock and regenerates the cache; every other concurrent request for the same key waits briefly, then reads the now-fresh entry instead of also hitting the DB. The design question becomes "how do I elect a single leader to rebuild this key," not "what happens on a miss."
• Probabilistic early expiration: refresh the key before it actually expires, with rising probability as expiry approaches — regeneration happens off the critical path, with no hard cliff.
• Never let TTL be the only expiry mechanism for hot keys — actively refresh them in the background via cron/worker so they never truly miss.
• Locking with a short grace period: hold a mutex per key during regeneration; other readers serve stale data for a few hundred ms rather than blocking on the DB.

value = cache.get(key)
if value is not None:
    return value

if acquire_lock(key, ttl=500ms):        # only one caller wins this
    try:
        value = db.query(...)
        cache.set(key, value, ttl=jittered(base_ttl))
        return value
    finally:
        release_lock(key)
else:
    # someone else is already rebuilding — wait briefly, then re-read
    sleep(50ms)
    return cache.get(key) or stale_cache.get(key)

04Lock contention

Multiple threads or processes need exclusive access to the same resource — a row, a data structure, a critical section — and end up serializing behind a lock. As concurrency rises, more time is spent waiting for the lock than doing useful work.

Why it happens

• Coarse-grained locks — locking an entire table or object when only a small part needs protection.
• Long-held locks — a transaction holding a row lock while doing slow, unrelated work like an external API call.
• High contention on a small number of hot rows — everyone incrementing the same counter, many transactions updating the same inventory record.
• In application code: synchronized blocks or mutexes guarding shared in-memory state.

Diagnosis

• Thread dumps show many threads BLOCKED waiting on the same monitor/lock.
• Database: SHOW ENGINE INNODB STATUS (MySQL) or pg_locks (Postgres) reveals long lock-wait chains and deadlock victims.
• CPU utilization can look low even though latency is high — a classic tell that the bottleneck is waiting, not computing.
• Throughput plateaus or drops even as you add more threads or instances — Amdahl's Law in action, the serialized portion dominates.

Fix / prevention

• Reduce lock scope and hold time: do slow work (I/O, external calls) outside the critical section.
• Move from pessimistic to optimistic concurrency control where contention is high but conflicts are actually rare — versioned rows / compare-and-swap instead of SELECT ... FOR UPDATE.
• Shard the contended resource: instead of one counter row, use N counter shards and sum them (same principle as hot partitions, below).
• Use appropriate isolation levels: don't default to the strictest level everywhere — know your actual consistency requirement.
• Queue-based serialization: if a resource genuinely must update one-at-a-time, make that explicit via a queue rather than many threads fighting over a lock.

05Hot partitions

In a sharded or partitioned data store (DynamoDB, Cassandra, sharded Postgres/MySQL), data spreads across nodes by hashing a partition key. A hot partition happens when a disproportionate share of traffic lands on one partition — because one key, or a small set of keys, is far more popular than the rest — even though the cluster as a whole has spare capacity.

Why it happens

• Low-cardinality keys: partitioning by something like status = "ACTIVE" or a date-only value concentrates huge volume on very few partition values.
• Celebrity effect: one user, one product, one leaderboard entry gets orders of magnitude more reads/writes than everyone else.
• Time-series data: keying by deviceId or a fixed date bucket means all of "today's" writes land on the same physical partition, since time-ordered data inherently targets one logical key repeatedly.
• Per-partition limits are hard limits — in DynamoDB, a single partition tops out around 3,000 RCU / 1,000 WCU regardless of how much total capacity the table has provisioned, so a hot partition throttles even when the table "looks" underutilized.

Diagnosis

• Throttling errors (e.g. ProvisionedThroughputExceededException) even though aggregate table/cluster metrics show spare capacity.
• Per-key traffic visibility tooling (CloudWatch Contributor Insights for DynamoDB, or equivalent) to identify which keys are hot before fixing anything blindly.

Fix / prevention

• Write sharding: append a random or hashed suffix to the hot key (user#123 → user#123#shard3) so writes spread across N physical partitions; fan out reads across all shards and aggregate.
• Composite keys with higher cardinality: combine a low-cardinality attribute with a high-cardinality one — "ACTIVE#customer123" instead of just "ACTIVE".
• Shard selectively: you don't need to shard the whole keyspace — only the specific hot entities (e.g. the top 100 leaderboard users), leaving the long tail on a simple single-key design.
• Cache in front of the hot partition (DAX/Redis) so the partition itself only sees a fraction of true read volume.
• Time-bucketing for time-series: roll into new partition keys per time window (per hour/day) instead of one key growing forever.
• Built-in mechanisms like DynamoDB Adaptive Capacity help automatically, but take several minutes to kick in — not fast enough to save you from a sudden spike. Proactive key design still matters.

// write: spread across N shards
shard = random(0, N-1)
put_item(pk=f"user#{userId}#shard{shard}", ...)

// read: fan out and merge
results = [get_item(pk=f"user#{userId}#shard{i}") for i in range(N)]
merged = merge(results)

06Slow queries

The most direct root cause on this list — a query that takes far longer than it should, tying up connections, locks, and CPU/IO on the database, and cascading into every other failure mode above.

Why it happens

• Missing or unused indexes, causing full table scans.
• N+1 query patterns from ORMs — one query for a list, then one query per item for related data.
• Poor query plans from stale statistics.
• Large, unbounded result sets — SELECT * with no pagination or limit.
• Lock waits disguised as slow queries — the query itself is fast, but it's queued behind another transaction's lock.
• Open-session-in-view (common in Spring) keeping a DB transaction open for the entire HTTP request lifecycle, including slow downstream calls — so otherwise-fast queries hold connections far longer than necessary.

Diagnosis

• Slow query logs — slow_query_log in MySQL, log_min_duration_statement in Postgres.
• EXPLAIN ANALYZE to check for sequential scans where an index scan was expected.
• APM traces showing time spent in DB calls vs. application logic.
• Correlate a rise in average query duration with the rest of the cascade — very often the actual root cause behind "pool exhausted" or "lock wait timeouts spiked."

Fix / prevention

• Add/verify indexes matching actual query predicates and sort orders.
• Fix N+1 patterns with eager loading or batched fetches.
• Set query timeouts so one bad query can't monopolize a connection indefinitely.
• Paginate everything; never return an unbounded result set.
• Disable open-session-in-view where it's not needed; keep transactions as short as possible.
• Use read replicas to offload read-heavy queries away from the primary.

07Network bottlenecks

Latency or throughput limits introduced by the network path between services, not the services themselves: bandwidth saturation, DNS resolution delays, TLS handshake overhead, cross-region/cross-AZ hops, load balancer or proxy limits.

Why it happens

• Chatty microservices making many small synchronous calls instead of batching.
• Cross-region calls adding tens to hundreds of ms of unavoidable latency per hop.
• Load balancer health checks or connection limits misconfigured — healthy instances marked unhealthy, or connections dropped.
• NIC/bandwidth saturation on a single host under heavy fan-out — broadcasting to many downstream services at once.
• DNS caching issues causing repeated, slow lookups under load.

Diagnosis

• Compare "time in app code" vs. "time in transit" using distributed tracing — a trace waterfall showing large gaps between spans.
• Load balancer metrics: active connections, backend error rate, health check failures.
• ping / traceroute / mtr for raw latency and packet loss where you can debug directly.
• Rule out DNS — slow or failing resolution often masquerades as a generic "connection timeout."

Fix / prevention

• Reduce cross-service chattiness: batch requests, use gRPC/streaming instead of many round trips.
• Co-locate latency-sensitive services in the same AZ/region.
• Tune load balancer health check intervals/thresholds so transient blips don't cause mass instance ejection.
• Use connection keep-alive and HTTP/2 multiplexing to avoid repeated handshake overhead.
• Circuit breakers and timeouts on every outbound call, so one slow network path doesn't stall the whole request chain.

08GC pauses

In managed-memory runtimes (JVM, .NET CLR, Go's GC), the garbage collector periodically pauses application threads ("stop-the-world") to reclaim memory. Under memory pressure or with a poorly tuned collector, these pauses grow long and frequent enough to show up as latency spikes — and in extreme cases, cause the process to be marked unhealthy by orchestration or a load balancer.

Why it happens

• Heap sized too small for the actual working set, forcing frequent collections.
• High allocation rate — lots of short-lived object churn overwhelming the young generation.
• Memory leaks — objects that should be garbage but are still referenced, growing the old generation until a full GC is needed.
• An old/default collector (Parallel GC) used for a latency-sensitive service instead of a low-pause one (G1, ZGC, Shenandoah).
• Large heaps paradoxically making full-GC pauses worse if not tuned for concurrent/incremental collection.

Diagnosis

• GC logs (-Xlog:gc* on modern JVMs) showing pause duration and frequency.
• Correlate p99/p999 latency spikes with GC pause timestamps.
• Heap dumps / profilers (async-profiler, VisualVM) to find allocation hotspots or leaks.
• Kubernetes liveness-probe failures that line up exactly with long GC pauses — the process is "alive" but unresponsive during STW.

Fix / prevention

• Choose a low-pause collector for latency-sensitive services — G1 as a solid default, ZGC/Shenandoah for very large heaps needing sub-millisecond pauses.
• Right-size the heap: too small causes GC thrashing, too large without a concurrent collector causes long full-GC pauses.
• Reduce allocation pressure — object pooling on very hot paths, avoiding unnecessary boxing, streaming instead of buffering large payloads.
• Fix leaks — common culprits are unbounded caches, listeners/callbacks never unregistered, ThreadLocals never cleared.
• Set liveness-probe thresholds with GC behavior in mind, so a normal (if long) pause doesn't trigger an unnecessary pod restart that compounds the problem.

-Xms4g -Xmx4g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:+ParallelRefProcEnabled
-Xlog:gc*:file=/var/log/app/gc.log:time,level,tags

09The interview diagnostic checklist

When an interviewer says "the service is slow, walk me through debugging it," this is the mental checklist worth narrating out loud, roughly in order:

1. CPU check karoIs CPU actually maxed out, or low despite high latency? Low CPU + high latency usually means waiting — locks, I/O, network — not computing.
2. Memory check karoIs the process near its memory limit? Climbing steadily (a leak) or spiking in bursts (an allocation storm)?
3. Garbage collection pauses dekhoCorrelate GC pause timestamps with latency spikes — often the fastest way to confirm or rule out GC as the culprit.
4. Network latency verify karoUse distributed tracing to see how much time is in the network vs. application code. Check cross-AZ/region hops specifically.
5. Load balancer healthy hai ya nahiCheck LB-level metrics — are backend instances flapping between healthy and unhealthy? Are health check failures correlated with GC pauses or pool exhaustion, i.e. is the LB reacting to a downstream problem rather than causing one?

10How these failures cascade

In a real incident, these almost never happen in isolation. A typical chain:

This is why a strong interview answer names the failure mode you'd check first, but also explains how you'd separate root cause from downstream symptom — for example: "pool exhaustion is often not a pool-sizing problem, it's a symptom of slow queries or lock contention upstream."

11References

This guide draws on official documentation (AWS, Oracle/OpenJDK, PostgreSQL, MySQL) and independent engineering write-ups. For interview practice specifically, cross-reference against more than one prep source and real interview-experience threads — question patterns and interviewer expectations vary by company and change over time.