Load Balancers

Scaling the Web#

When an application outgrows one server, the answer is horizontal scaling: run several identical servers. That raises an immediate question: which address does DNS give to users? The answer is a load balancer. DNS points at the balancer, the balancer accepts every connection, and it spreads the work across a pool of healthy backends. Done well, it also gives you zero-downtime deploys and automatic removal of failed servers. Done badly, it becomes the source of mysterious 502s and uneven load.

Layer 4 vs. Layer 7#

Layer 4 (transport) load balancing works on TCP/UDP connections. It chooses a backend using the IP addresses and ports, then forwards packets (or proxies the byte stream) without understanding the contents.

• Pros: very fast, protocol-agnostic (databases, MQTT, any TCP or UDP service), TLS can pass straight through to backends.
• Cons: cannot route by URL, header or cookie; one long-lived connection always stays on one backend.
• Examples: Linux IPVS, HAProxy in mode tcp, cloud network load balancers.

Layer 7 (application) load balancing terminates the client connection, usually decrypts TLS, parses the HTTP request and opens (or reuses) a separate connection to the chosen backend.

• Pros: route by host, path, header or cookie; retry failed requests; add headers; compress; balance each HTTP/2 or gRPC request individually.
• Cons: more CPU (TLS and parsing), must understand the protocol, and the backend no longer sees the client's IP directly.
• Examples: NGINX, HAProxy in mode http, Envoy, cloud application load balancers.

Large platforms often stack both: an L4 tier (sometimes using anycast and direct server return, where backends reply to clients directly and bypass the balancer on the return path) in front of a fleet of L7 proxies.

Load Balancing Algorithms#

• Round robin: each backend in turn. Fine when servers and requests are uniform.
• Weighted round robin: proportional turns, for fleets mixing instance sizes.
• Least connections: the backend with the fewest active connections. Better when request durations vary widely or connections are long-lived (WebSockets).
• Least response time / least outstanding requests: accounts for how fast each backend is currently answering.
• Power of two random choices: pick two backends at random, send to the less loaded one. Nearly as good as global least-connections, but needs no shared view of all servers, which is why distributed proxies such as Envoy offer it.
• Hashing (by client IP, header or URL): the same key always lands on the same backend, useful for cache locality and stickiness.

A small simulation makes the behaviour concrete, including why the kind of hashing matters:

import bisect, hashlib, itertools

def h(key: str) -> int:
    return int.from_bytes(hashlib.sha256(key.encode()).digest()[:8], "big")

class RoundRobin:
    def __init__(self, servers): self.cycle = itertools.cycle(servers)
    def pick(self, key=None): return next(self.cycle)

class LeastConnections:
    def __init__(self, servers): self.active = {s: 0 for s in servers}
    def pick(self, key=None):
        server = min(self.active, key=self.active.get)
        self.active[server] += 1          # caller must decrement when the request ends
        return server
    def done(self, server): self.active[server] -= 1

class ModHash:
    def __init__(self, servers): self.servers = list(servers)
    def pick(self, key): return self.servers[h(key) % len(self.servers)]

class ConsistentHash:
    def __init__(self, servers, vnodes=100):
        self.ring = sorted((h(f"{s}#{i}"), s) for s in servers for i in range(vnodes))
        self.points = [p for p, _ in self.ring]
    def pick(self, key):
        i = bisect.bisect(self.points, h(key)) % len(self.ring)
        return self.ring[i][1]

rr = RoundRobin(["app1", "app2", "app3"])
print("round robin:", [rr.pick() for _ in range(5)])

lc = LeastConnections(["app1", "app2", "app3"])
a, b = lc.pick(), lc.pick()
lc.done(a)                                 # first request finished
print("least conn: ", [a, b, lc.pick()])

clients = [f"client-{i}" for i in range(10_000)]
before, after = ["app1", "app2", "app3", "app4"], ["app1", "app2", "app3"]   # app4 removed
for cls in (ModHash, ConsistentHash):
    x, y = cls(before), cls(after)
    moved = sum(x.pick(c) != y.pick(c) for c in clients)
    print(f"{cls.__name__:15} clients remapped after removing app4: {moved / len(clients):.0%}")

round robin: ['app1', 'app2', 'app3', 'app1', 'app2']
least conn:  ['app1', 'app2', 'app1']
ModHash         clients remapped after removing app4: 74%
ConsistentHash  clients remapped after removing app4: 21%

With naive hash % N, removing one of four servers reshuffles about three-quarters of all clients, destroying every cache and sticky session at once. Consistent hashing places servers at many points ("virtual nodes") on a ring and only moves the keys that belonged to the removed server, roughly a quarter here. That property is why caches, sharded stores and hash-based balancers use it.

Sticky Sessions#

Session affinity sends a client back to the same backend, by source-IP hash or, more reliably, a balancer-issued cookie. It is needed when servers keep session state in local memory. The costs: uneven load (one heavy user pins one server), lost sessions when that server dies, and many users behind one corporate NAT or CGNAT address all hashing to one backend. The better fix is to make servers stateless by moving sessions to a shared store; see Cookies & HTTP Sessions.

Configuring a Real Balancer#

NGINX as an L7 balancer with passive health checking:

upstream app {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 weight=2;           # bigger instance
    keepalive 32;                             # reuse upstream connections
}

server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";       # required for upstream keepalive
        proxy_next_upstream error timeout;    # retry another server on connect errors
    }
}

Open-source NGINX only marks servers failed passively, after real requests fail. HAProxy performs active checks:

backend app
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    server app1 10.0.1.11:8080 check inter 2s fall 3 rise 2
    server app2 10.0.1.12:8080 check inter 2s fall 3 rise 2

inter 2s fall 3 rise 2 means: check every two seconds, remove after three consecutive failures, re-add after two successes. The rise value prevents a flapping server from bouncing in and out of the pool.

Health Checks That Tell the Truth#

A balancer is only as good as its health checks. Distinguish two questions:

• Liveness: is the process alive? If not, restart it.
• Readiness: can it serve traffic right now? If not, stop sending it requests.

A check that only returns "OK" from the web framework will keep routing to an instance whose database pool is exhausted. But a "deep" check that fails whenever a shared dependency is down is also dangerous: if the database blips, every instance fails its check simultaneously and the balancer has no healthy targets. Good practice is to check local, instance-specific conditions (can I reach my dependencies with a short timeout, is my disk full, am I still warming up) and to rely on the fact that many balancers fail open, routing to all targets when all are unhealthy, rather than serving nothing.

Connection Draining and Deploys#

When an instance is removed for a deploy, requests in flight on it must finish. Connection draining (deregistration delay) stops new traffic to the instance while letting existing connections complete for a grace period. On the application side, handle SIGTERM by failing the readiness check first, waiting for the balancer to notice, finishing in-flight work, and only then exiting. Skipping this is why many teams see a burst of 502s on every deploy.

Common Failure Modes#

The keep-alive race in the first row is one of the most common causes of unexplained 502s behind cloud load balancers. Header handling for client IPs is covered in Reverse Proxies and API Gateways.

Global Server Load Balancing#

A single balancer is still one location and one failure domain. GSLB steers users across regions using DNS (answering with the nearest or healthiest region's address) or anycast (announcing the same IP from many locations and letting internet routing choose). DNS-based steering is limited by resolver caching and TTLs, so failover takes at least as long as the TTL; anycast reacts at routing speed but needs control of BGP. CDNs apply the same ideas at the edge, and DNS explains the TTL behaviour that bounds failover time.

Practice#

Run three copies of python -m http.server on ports 8081-8083 in different directories (each with an index.html naming the server), put the NGINX config above in front with its server lines pointed at 127.0.0.1:8081-8083 (first with the default round robin, then with least_conn), and hit it with for i in $(seq 9); do curl -s localhost; done. Kill one backend and observe how many requests fail before NGINX marks it down.