Load Balancing

In a Nutshell

A load balancer sits in front of a pool of servers and spreads incoming requests across them, so no single server is overwhelmed while others sit idle. It's the linchpin of horizontal scaling: without it, adding servers does nothing because clients wouldn't know they exist. Beyond distribution, load balancers provide health checking (route only to healthy nodes), failover (pull dead nodes out automatically), and often TLS termination, session affinity, and traffic shaping. They operate at either the transport layer (L4 — fast, connection-level) or the application layer (L7 — smart, content-aware routing).

How It Actually Works

L4 vs L7 Load Balancing

Aspect	L4 (Transport)	L7 (Application)
Operates on	TCP/UDP connections, IP:port	HTTP requests, URLs, headers, cookies
Routing decisions	Based on network info only	Based on content (path, host, method)
Speed	Very fast, low overhead	Slower — must parse the request
Capabilities	Simple distribution	Path routing, header rewrites, A/B, sticky sessions, WAF
TLS	Passes through (or terminates)	Usually terminates and inspects
Examples	AWS NLB, HAProxy (TCP mode), IPVS	AWS ALB, NGINX, Envoy, Traefik

Rule of thumb: use L4 when you need raw throughput and protocol-agnostic forwarding; use L7 when you need content-based routing (e.g., /api → service A, /images → service B) or HTTP-aware features.

Load Balancing Algorithms

Algorithm	How It Works	Best For
Round Robin	Cycle through servers in order	Homogeneous servers, uniform requests
Weighted Round Robin	Round robin biased by server capacity weights	Mixed instance sizes
Least Connections	Send to the server with fewest active connections	Long-lived/variable-duration requests
Least Response Time	Fewest connections + lowest latency	Latency-sensitive services
IP Hash / Consistent Hash	Hash client key → same server each time	Session affinity, cache locality
Random (Power of Two Choices)	Pick 2 at random, send to the less loaded	Great distribution, low coordination

Power of Two Choices deserves a callout: picking two servers at random and choosing the less loaded one yields nearly optimal balancing with almost no coordination — it's what many modern proxies (and JavaScript-free service meshes) use.

Health Checks: The Feature That Actually Matters

Distribution is easy; not sending traffic to broken servers is the real value. Two flavors:

• Passive — observe real traffic; if a server returns errors or times out, mark it unhealthy.
• Active — periodically probe an endpoint (e.g., GET /healthz) and route only to responders.

Active health check loop:
  every 5s:  GET /healthz  →  200 OK   →  keep in rotation
                            →  timeout/500 → remove from pool
                            (require N consecutive failures to avoid flapping)
  when recovered: require M consecutive successes before re-adding

Two subtleties: distinguish liveness (is the process up?) from readiness (can it serve traffic right now? — deps warm, not draining). And use hysteresis (N failures to eject, M successes to re-add) so a single blip doesn't flap a node in and out.

Session Affinity ("Sticky Sessions")

If a server keeps per-user state locally, the LB can pin a user to the same server via a cookie or IP hash. This is a crutch — it undermines even distribution and breaks when the server dies. Prefer stateless services with shared state (Redis) so any server can handle any request. Use stickiness only when you can't avoid local state.

Avoiding the LB as a Single Point of Failure

The load balancer must not itself be a SPOF. Techniques:

• Redundant LBs in active-active or active-passive with a floating/virtual IP (VRRP).
• DNS-based distribution across multiple LB endpoints.
• Global Server Load Balancing (GSLB) / Anycast for multi-region traffic steering.
• Cloud managed LBs (ALB/NLB) are themselves horizontally scaled and multi-AZ by default.

Seeing It in Action

Scenario: NGINX as an L7 load balancer with health checks and least-connections.

upstream app_servers {
    least_conn;                          # send to the least busy server
    server 10.0.1.10:8080 weight=2;      # bigger box gets 2x traffic
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
    server 10.0.1.13:8080 backup;        # only used if others are down
}

server {
    listen 443 ssl;
    server_name api.example.com;
    # ... TLS config (LB terminates TLS here) ...

    location /api/ {
        proxy_pass http://app_servers;
        proxy_next_upstream error timeout http_502 http_503;  # retry elsewhere
        proxy_connect_timeout 2s;
    }

    location = /healthz {                # LB's own health endpoint
        access_log off;
        return 200 "ok\n";
    }
}

What's happening: requests to /api/ are TLS-terminated at the LB, then routed to the least-busy backend (weighted for a larger box). If a backend errors or times out, NGINX transparently retries the request on another server, and a backup server absorbs traffic only during a full outage.

Interview Questions

1. Q: What's the difference between L4 and L7 load balancing, and when would you use each? Hint: L4 routes on TCP/UDP + IP:port — fast, protocol-agnostic, no request inspection. L7 routes on HTTP content (path, host, headers, cookies) — enables content-based routing, TLS termination, A/B, WAF, at higher overhead. Use L4 for raw throughput and non-HTTP protocols; L7 for smart HTTP routing and features.

2. Q: Compare round robin, least connections, and consistent hashing. Hint: Round robin = simple even rotation, assumes uniform requests. Least connections = adapts to variable request durations by favoring less-busy servers. Consistent hashing = maps a client key to a stable server (session affinity, cache locality) and minimizes reshuffling when servers are added/removed.

3. Q: How does a load balancer detect and handle a failed server? Hint: Health checks — active (periodic probes to /healthz) and/or passive (observing real request failures). Use hysteresis (N consecutive failures to eject, M successes to re-add) to prevent flapping. On failure, remove from the pool and optionally retry the in-flight request on another node. Distinguish liveness from readiness.

4. Q: What are sticky sessions, and why are they generally discouraged? Hint: Session affinity pins a user to one server (via cookie/IP hash) because that server holds local state. Downsides: uneven load, lost sessions when the server dies, and harder scaling/deploys. Preferred alternative: make services stateless and store session state in a shared store (Redis/DB) so any server can serve any request.

5. Q: The load balancer itself can be a single point of failure. How do you address that? Hint: Run redundant LBs (active-active or active-passive) with a virtual/floating IP via VRRP; use DNS or Anycast to spread across multiple LB endpoints and regions (GSLB); or rely on cloud-managed LBs that are inherently multi-AZ and horizontally scaled. Health-check the LBs themselves.

References

• NGINX: What Is Load Balancing? — algorithms and configuration
• AWS Elastic Load Balancing — ALB (L7) vs NLB (L4) vs GWLB
• The Power of Two Random Choices (Mitzenmacher) — the theory behind two-choice balancing

Dive Deeper

• Envoy Proxy Architecture — a modern L7 proxy powering service meshes
• Google Maglev: A Fast and Reliable Software Network Load Balancer — how Google builds LBs at planetary scale
• Facebook Katran — XDP/eBPF-based L4 load balancing