Load balancing: L4 vs L7

For this lesson, we'll cover the L4 vs L7 split, algorithms, health checks, sticky sessions, draining, global and client-side balancing, and how the balancer avoids being the weak link. Interviewers ask about it constantly because "add a load balancer" is the first scaling move in nearly every design, and the follow-up ("which kind, which algorithm, what if it dies?") shows who has run one.

01Why load balancers exist

Four jobs, each building on the last:

1. Scale out. Ten stateless servers behind a balancer handle roughly ten times the traffic of one.
2. Survive failures. Health checks pull a dead instance, so a crash is a capacity dip, not an outage.
3. Change without downtime. Deploys, canaries and autoscaling add and remove pool members behind a stable address.
4. One front door. Clients know one name; the fleet behind it can change every minute.

02L4 vs L7: what the balancer can see

The layer number tells you how far into the packet the balancer looks, and that decides everything else. Here's the difference side by side.

An L4 balancer looks at addresses, ports and protocol (the 5-tuple), picks a backend when a connection starts, and forwards every packet of that connection to the same place. It doesn't decrypt anything. AWS's Network Load Balancer is the classic managed example; AWS documents it as working at layer 4 and handling millions of requests per second. An L7 balancer finishes the TCP and TLS handshakes itself, parses each HTTP request, and opens its own connections to backends, which makes it a reverse proxy.

The gRPC team's own guidance puts the trade-off neatly: pick L4 when latency or proxy cost matters most, and L7 when load varies a lot between connections, because only L7 can spread the streams of one HTTP/2 connection across several backends.

03Algorithms: how the next server is picked

There's no best algorithm, only the one that matches your traffic. This table is what I'd want a candidate to be able to reproduce.

The last one shows depth. If 20 balancer instances run "least connections" on stale counts, they all spot the same idle server and pile onto it. Sampling two at random and taking the better one keeps most of the benefit while making that herd very unlikely.

All of these are one line in nginx. The upstream docs list least_conn, ip_hash (which hashes the first three octets of an IPv4 address), hash $key consistent (ketama consistent hashing) and random two least_conn (power of two choices). Round robin is the default.

upstream orders_api {
    least_conn;                                   # or: random two least_conn;
    server 10.0.1.11:8080 weight=2;               # bigger box, twice the share
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 backup;                 # only used if the others are down
}

# For a cache tier, pin each key to a node instead:
# upstream cache_tier { hash $arg_key consistent; server ...; }

04Health checks: active vs passive

A balancer is only as good as its idea of which servers are alive. There are two ways to know.

• Active checks: the balancer probes each target on a schedule (TCP connect, or GET /health expecting a 200) and flips its state after a run of failures or successes.
• Passive checks: the balancer watches real traffic and ejects a server that keeps failing it. nginx's max_fails and fail_timeout work this way (defaults: 1 failure within 10 seconds marks the server unavailable for 10 seconds), and Envoy calls it outlier detection.

Use both: active checks catch a server before users hit it, passive checks react within a few requests. Make the health endpoint check what the instance needs to serve (can it reach its database?), but don't make it fail when a shared dependency is down, or every instance goes unhealthy at once and the balancer has nothing left to route to.

05Sticky sessions, and why to avoid them

Sticky sessions pin a user to one backend, usually with a cookie the balancer sets, to rescue apps that keep session state in memory. I treat them as a smell:

• Uneven load. A few heavy users stuck to one box, and you can't move them.
• Failures lose state. When that server dies, its users lose their sessions anyway.
• Scaling doesn't help existing users. New instances only get new sessions.
• Deploys drag. Draining waits on users who are pinned there.

The better default is stateless servers with sessions in a shared store such as Redis, or in a signed token. Affinity is fine as a performance hint (a user's data usually warm in one node's cache), not as a correctness requirement. WebSockets are sticky by nature, so plan for them to drop and reconnect.

06Connection draining

Removing a server on purpose shouldn't break its requests. Draining (AWS calls it deregistration delay) stops new requests to the target while in-flight ones finish; AWS waits 300 seconds by default. Set it a bit longer than your slowest normal request. For WebSockets and streams, drain for a while, then close them so clients reconnect elsewhere with backoff.

07Global vs regional load balancing

Everything so far happens inside one region. In front of that sits a second, coarser decision: which region should this user reach? Here's the full stack.

Large services combine them, then balance inside the region. Multi-region and active-active covers the data side.

08Client-side load balancing

Inside a microservice system, there's often no balancer box at all. Each client gets the instance list from service discovery and picks one per call: one hop fewer and no central bottleneck, at the cost of balancing logic in every client (or a sidecar such as Envoy).

gRPC is where this matters most. A gRPC client multiplexes all its calls over one long-lived HTTP/2 connection, so behind an L4 balancer every call from that client lands on one backend, and adding servers doesn't spread existing load. gRPC's default pick_first policy has the same effect. Switching to round_robin makes the client connect to every address the resolver returns and rotate per call:

ManagedChannel channel = ManagedChannelBuilder
        .forTarget("dns:///orders.internal:8443")   // resolver returns every instance
        .defaultLoadBalancingPolicy("round_robin")  // default is pick_first
        .build();

The alternatives are an HTTP/2-aware L7 balancer or a service mesh. The gRPC team recommends client-side balancing for trusted, high-traffic clients and a proxy when you need a trust boundary.

09The load balancer as a single point of failure

If every request goes through one balancer, you've moved the single point of failure, not removed it. Two patterns fix that:

• Active-passive pair: two balancers share a floating virtual IP (VRRP, for example via keepalived); the standby takes the IP when the active one stops sending heartbeats. Simple, but half the hardware idles and the pair's capacity is one box.
• Active-active tier: many balancer machines, all serving. Upstream routers spread flows across them with ECMP, or anycast spreads them across sites. Losing one costs 1/N of capacity.

Managed cloud balancers do this for you: AWS documents that enabling an Availability Zone creates a load balancer node in that zone, and the balancer's DNS name resolves to the nodes. Your job is to enable at least two zones and keep enough headroom in each.

10Practise: pick the layer

Here's how I'd answer out loud.

11How it shows up in design problems

• Design a URL shortener: stateless app servers behind an L7 balancer with round robin; the redirect path is short and uniform.
• Design WhatsApp: millions of long-lived WebSocket connections, so least connections, draining plans and a mapping of which server holds which user's connection.
• Design an LLM gateway: request cost varies by orders of magnitude, so least outstanding requests or power of two choices beats round robin.
• Design a distributed cache: consistent hashing on the key, so a node change only moves a slice of the cache.
• Design a rate limiter: hash the user or API key to one limiter node so its counter isn't split across many.

At this point, you should ask your interviewer whether servers hold any per-user state. That one answer decides between round robin, hashing and sticky sessions.

12Interview follow-ups

What if the interviewer asks why not use DNS round robin instead of a load balancer?

DNS can spread users across a few addresses, but it can't react quickly: answers are cached until the TTL expires, clients often stick to the first address, and DNS has no idea whether a server is overloaded. It's a good fit for picking regions or balancer nodes, and a poor fit for picking individual app servers.

What if a new instance joins and immediately gets flooded?

That's least connections treating an empty server as the best target, while its caches and JIT are still cold. Use slow start (ramp its weight up over a minute or two) or power of two choices, and make readiness checks pass only once it's warmed up.

What if your health checks all pass but users are seeing errors?

The check is too shallow, for example it returns 200 whenever the process is up. Add passive checks that eject servers returning 5xx on real traffic, and make the active check exercise what a request needs. Then check whether the errors come from a shared dependency, which no amount of rerouting will fix.

What if the whole region's load balancer tier fails?

Global balancing takes over: DNS health checks or an anycast route withdrawal send users to another region. How fast depends on DNS TTLs or BGP convergence, and the other region needs headroom to absorb the traffic. Without that headroom, failing over just moves the outage.

13How to say it in the interview

When I ask about load balancing, I'm listening for four things: that you pick a layer for a reason, that your algorithm matches the traffic, that you know how unhealthy servers get removed, and that the balancer itself isn't a single box.