Opening Scenario
Your API service handles 500 requests/second without issue. You deploy a critical feature, traffic spikes to 3,000 requests/second, and your single server starts returning 502 errors. You spin up three more identical servers, but the errors don't stop — all the traffic is still hitting the original server. Something has to decide which server handles each request.
Question Progression
1. How do you distribute traffic across multiple servers?
Question: You have four backend servers, each capable of handling 1,000 req/s. How do you actually route an incoming request to one of them instead of all going to server 1?
Reasoning: A client makes a request to a single IP address or hostname. It has no knowledge of your server fleet. Something upstream must intercept that request and forward it to one of your available servers. The client never needs to know which backend handled…
In the full tutorial
- Closing: When to Reach for This