Load Balancing (L4 vs L7, Round-Robin, Consistent Hashing)

1. What Is It?

A load balancer distributes incoming requests across a pool of servers to maximize throughput, minimize latency, and ensure high availability. Without a load balancer, a single server handles all traffic — it becomes a performance bottleneck and a single point of failure. With a load balancer, horizontal scaling becomes possible: add servers to the pool, the load balancer automatically distributes work.

The two fundamental dimensions of load balancing are layer (L4 vs L7) and algorithm (how to choose a backend). Layer determines what the load balancer can see: L4 sees only IP/port; L7 sees the full HTTP request. The algorithm determines which server gets each request: round-robin for simplicity, least connections for variable workloads, consistent hashing for session affinity. These choices have significant performance, cost, and operational implications.

2. How It Works

L4 Load Balancing (Transport Layer)

L4 load balancers operate on TCP/UDP packets, reading only source/destination IP addresses and ports. They cannot inspect HTTP headers, URLs, or cookies.

NAT mode: The load balancer rewrites the destination IP of inbound packets and forwards to a backend. Return traffic flows back through the load balancer (it must rewrite the source IP back to the VIP). Bottleneck: both request and response traffic passes through the LB.

Direct Server Return (DSR): The load balancer only rewrites the destination MAC address. The backend processes the request and responds directly to the client, bypassing the load balancer entirely on the return path. This is why DSR can handle 100+ Gbps: the LB only touches small inbound request packets; the large response (e.g., video stream, large file download) goes straight from backend to client.

NAT mode:
Client → LB (rewrite dest IP) → Backend → LB (rewrite src IP) → Client
         LB handles ALL traffic (bottleneck for response-heavy workloads)

DSR mode:
Client → LB (rewrite dest MAC) → Backend ──→ Client (direct)
         LB only touches inbound requests — responses bypass LB entirely

Use cases: Raw TCP/UDP throughput, gaming servers, DNS, IoT, database connection load balancing. Cannot route by URL or user identity.

L7 Load Balancing (Application Layer)

L7 load balancers terminate the client TCP connection, parse the full HTTP request, and make routing decisions based on content. They then open a new connection to the selected backend.

Routing capabilities:

• Path-based: /api/* → API cluster; /static/* → CDN cluster; /admin/* → internal cluster
• Header-based: Route by User-Agent (mobile vs desktop), Accept-Language, custom headers
• Cookie-based: Sticky sessions — route a user to the same backend using a session cookie
• Host-based: api.example.com → API servers; www.example.com → web servers
• HTTP method-based: GET → read replicas; POST/PUT → write primaries

Cost: TLS termination + HTTP parsing adds 5–20ms latency and requires CPU. But SSL termination is a prerequisite for L7 routing — you can't read HTTP content without decrypting it first.

-+ResetFullscreen

Load Balancing Algorithms

1. Round-Robin

Rotate requests sequentially through servers: A → B → C → A → B → C...

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A (cycle repeats)

• Pros: Simple, stateless, even distribution for uniform request sizes
• Cons: Ignores server load, capacity, and request duration. A server handling a slow 5-second request gets new requests at the same rate as an idle server.
• Best for: Homogeneous servers, short-lived stateless requests, similar request durations

2. Weighted Round-Robin

Assign weights proportional to server capacity. Servers get requests in proportion to their weight.

Server A: weight=5, Server B: weight=2, Server C: weight=1
Cycle: A A A A A B B C (then repeat)

• Best for: Heterogeneous server fleet (e.g., some instances are 4x core, others are 2x core)

3. Least Connections

Route each new request to the server with the fewest currently active connections.

Server A: 10 active connections
Server B: 2 active connections
Server C: 7 active connections
→ New request → Server B

• Better than round-robin when: Connections are long-lived (WebSockets, gRPC streams, DB connections) or request duration varies widely. Round-robin ignores the accumulated load from slow, long-running requests; least connections adapts in real time.
• Cons: Requires tracking connection state per server; adds slight overhead.

4. IP Hash (Session Stickiness)

Hash the client's source IP address to deterministically select a backend. All requests from the same IP always go to the same server.

hash("192.168.1.1") % 3 = 2 → Server C (always, until pool changes)

• Use case: When the application stores session state in memory on the server (not recommended, but common in legacy apps). Sticky sessions avoid session loss.
• Problems: (1) Mobile clients on dynamic IPs lose stickiness on IP change; (2) NAT-ed corporate networks where all employees share one IP → one server gets all their traffic; (3) Uneven load distribution if one IP is very active.
• Better alternative: Store session state in a shared Redis cache instead of pinning users to servers.

5. Consistent Hashing

Hash both the server IDs and the client/request key onto a ring (0–2³²). Each request maps to the nearest server clockwise on the ring. When a server is added or removed, only the requests that mapped to that server are remapped — all others are unaffected.

Ring: [0 .... A .... B .... C .... 2³²]
      request r1 → between 0 and A → maps to A
      request r2 → between A and B → maps to B
      Remove server A → r1 now maps to B (only r1 is affected)

• Use case: Session persistence with minimal disruption during scale events. Used in distributed caches (where consistent hashing determines which cache node holds a key) and CDN routing.
• Virtual nodes: Each physical server gets multiple positions on the ring, providing more even load distribution.

Algorithm Comparison

Full Architecture: L4 + L7 Layered

-+ResetFullscreen

Health Checks

Active health checks: The load balancer proactively sends synthetic probe requests (HTTP GET /health) to each backend at a fixed interval (e.g., every 30 seconds). If a backend fails N consecutive probes, it's marked unhealthy and removed from rotation. New requests are no longer sent to it.

Passive health checks: The load balancer monitors real client traffic. If a backend returns too many 5xx errors or times out too frequently, it's marked unhealthy. A real client request must fail before detection — suitable only for lower-criticality services.

For production, active health checks are standard. The /health endpoint should check both the server process AND its dependencies (DB connectivity, cache availability) to detect partial failures.

SSL Termination

Client ──HTTPS──▶ Load Balancer ──HTTP──▶ Backend Servers
                 (decrypts TLS)          (plain internal traffic)

SSL termination at the LB provides:

1. CPU offload: TLS handshake is CPU-intensive; backends are freed from this work
2. Centralized certificate management: Certificates on one LB, not on every backend
3. Prerequisite for L7 routing: The LB must read HTTP content to route by URL — requires decryption first
4. WAF and inspection: Decrypted traffic can be inspected for malicious patterns

Security note: traffic between LB and backends traverses the internal network unencrypted. For PCI-DSS/HIPAA compliance, use SSL passthrough or end-to-end TLS (re-encrypt from LB to backends with an internal cert).

Connection Draining (Zero-Downtime Deployments)

When a backend is removed from rotation (e.g., rolling deployment):

1. LB stops sending new requests to the draining instance
2. In-flight requests already processing are allowed to complete
3. After all active connections close (or timeout expires, e.g., 300s on AWS), instance is deregistered
4. The application must handle SIGTERM gracefully — finish in-flight work before exiting

3. Variants & Comparisons

AWS Load Balancers

Nginx vs HAProxy

4. When to Use It (and When NOT To)

Use L4 When:

• Raw TCP/UDP throughput is paramount (gaming, DNS, live streaming media)
• Static IP address required (NLB provides per-AZ static IPs)
• No content-based routing needed
• Minimizing latency overhead is critical (L4 adds sub-millisecond latency)

Use L7 When:

• URL path, host, or header-based routing is required
• TLS termination and certificate management should be centralized
• WebSocket / gRPC connection handling, HTTP/2 multiplexing
• WAF, rate limiting, or authentication at the gateway layer

Algorithm Selection:

• Round-robin: Default choice for stateless microservices with uniform request durations
• Least connections: Long-lived connections (WebSockets, gRPC, DB connections) or variable request duration
• Consistent hashing: Distributed caches (Memcached, Redis cluster) where cache hit rate depends on routing consistency; CDN origin selection
• IP hash / sticky sessions: Only as a last resort for legacy session-stateful apps. Prefer shared Redis session storage instead.

5. Real-World Usage

AWS ALB + NLB Combination (Common Pattern)

Large-scale AWS deployments often use NLB at the edge for a static IP (required for DNS-based global routing and allowlists) + ALB behind it for content-based routing. NLB adds sub-millisecond overhead; ALB handles path routing, SSL termination, and WAF. This layered approach combines the advantages of both.

Facebook / Meta — Load Balancing at Scale

Meta's load balancer (Proxygen + Katran) uses L4 ECMP (Equal Cost Multi-Path routing at the network layer) for raw distribution, then L7 at the application layer for content routing. Katran uses eBPF/XDP in the Linux kernel for line-rate L4 packet processing — enabling multi-hundred-Gbps throughput per machine without kernel network stack overhead.

Cloudflare — Anycast as Natural Load Balancing

Cloudflare uses anycast routing at the DNS/IP layer as an implicit first stage of load balancing: any request to Cloudflare's IP space is routed to the nearest PoP by BGP routing policies. Within each PoP, consistent hashing distributes requests across edge servers. This geographic distribution inherently balances load across ~300 PoPs globally without any centralized load balancer.

6. Interview Cheat Sheet

5 Sentences to Show Deep Understanding

1. "The key difference between L4 and L7 load balancing is visibility: L4 sees only IP:port and makes blind routing decisions, while L7 terminates TLS, reads the full HTTP request, and can route based on URL path, headers, cookies, and user identity — but at the cost of 5–20ms additional latency and CPU for TLS offload."

2. "Least connections outperforms round-robin for WebSocket, gRPC, and database connection pools because it accounts for accumulated connection load — round-robin treats a server handling 100 long-lived WebSocket connections the same as an idle server."

3. "SSL termination at the load balancer is not just a performance optimization — it's a prerequisite for L7 content-based routing, since the LB must read decrypted HTTP headers to route by URL path or cookie."

4. "Consistent hashing minimizes cache key remapping when the backend pool changes: instead of remapping all keys on a pool change (like modulo-based hashing), only the keys that mapped to the added/removed server are redistributed — critical for cache hit rate during scaling events."

5. "Zero-downtime deployments require both the load balancer (connection draining — stop new requests, let in-flight complete) AND the application (SIGTERM handler — finish current work before exiting) to cooperate; the LB alone cannot guarantee zero dropped requests if the app exits immediately."

Common Follow-Up Questions

Q: What is the thundering herd problem during a deployment, and how does load balancing help? A: When new instances spin up, they have cold caches (empty Redis, cold JVM JIT). If the LB immediately sends full traffic to new instances, they're overwhelmed. Solution: (1) gradual traffic ramp (weighted round-robin, start new instances at weight=1, ramp to 10); (2) warm-up period (20–30 seconds of reduced traffic); (3) health checks that include a "warm" state before marking ready.

Q: What's the difference between sticky sessions via IP hash vs cookie-based sticky sessions? A: IP hash is stateless on the LB but breaks for mobile users (dynamic IPs) and corporate NAT (many users sharing one IP). Cookie-based stickiness (e.g., AWSALB cookie) is more reliable — the LB sets a cookie on the first response, subsequent requests carry the cookie, and the LB uses it to route to the same backend. More overhead (cookie parsing) but much more reliable stickiness.

Q: How would you handle a hot-spot on one backend when using consistent hashing? A: Add virtual nodes — each physical server gets 100–150 positions on the consistent hash ring. This provides more even distribution (a single server isn't responsible for 1/N of the ring; instead, it holds N/150 of 150 virtual positions which are more uniformly distributed). For extreme hot-spots, add a second load balancing stage: consistent hash to a group of servers, then least-connections within the group.

Q: What happens to in-flight requests when a server fails? A: If active health checks detected the failure, the LB already stopped routing new requests to it. In-flight requests on that server are dropped (connection reset). Clients should retry on failure (typically on 5xx or connection errors). At the LB level, some L7 balancers (like HAProxy with option redispatch) automatically retry failed requests on a different backend.

Connections to Other Building Blocks

• Consistent Hashing: The consistent hashing building block describes the ring algorithm used by load balancers for session affinity and by distributed caches for key routing.
• API Gateway & Reverse Proxy: Often co-located with the L7 load balancer. The LB distributes traffic; the API gateway handles auth, rate limiting, and protocol translation.
• Rate Limiter: Rate limiting is implemented as LB middleware or at the API gateway sitting behind the LB. The LB provides the enforcement point.
• CDN & Edge Caching: CDN edge nodes are load balanced internally using consistent hashing (which edge server handles a given URL) and anycast (which PoP handles a given client).
• Circuit Breaker: Circuit breakers and load balancers both protect backends from overload. LBs distribute across healthy backends; circuit breakers stop traffic to a failing downstream service entirely.