Load Balancing

A load balancer distributes incoming traffic across the servers in a cluster so no single machine gets overwhelmed. It's a key component of web architecture, and it exists to prevent a specific failure: if all traffic converges on one or a few machines, those machines overload — latency spikes, performance drops, and the nodes can go offline entirely. Spreading the requests avoids that mess.

The balancer also props up availability. If a server dies mid-request, the balancer simply routes future requests to the machines still running, so the service as a whole stays up. It acts as the single point of contact for every client request — clients talk to the balancer, and the balancer decides which server actually answers.

Balancers aren't only for the front door. They can sit in front of any application component — backend app servers, the database, a message queue — to spread request load evenly across the machines powering that specific component.

To route intelligently, a balancer must know which servers are actually alive — so it runs regular health checks. It keeps a real-time list of the machines that are up, and forwards requests only to those. A machine that's up is in-service; a machine that's down is out-of-service. When a server fails a health check, it's dropped from the list. When it recovers and passes again, the balancer adds it back and resumes routing to it.

Every machine on the web has a unique IP address, and DNS is the system that lets you reach one without memorizing it. IP (Internet Protocol) delivers data packets between machines using those addresses — something like 2001:db8:0:1234:0:567:8:1. Nobody can hold addresses like that in their head, so the Domain Name System (DNS) maps a memorable domain name such as amazon.com to its IP address for you.

When you type a URL and hit enter, that event is called DNS querying. Resolving the name involves four kinds of servers working in a chain:

Here's the full lookup for amazon.com, step by step.

1. Browser → Resolver. You hit enter; the browser sends the request to the DNS Resolver (recursive nameserver), typically managed by your ISP.
2. Resolver → Root. The Resolver asks the Root nameserver where to find the right TLD server. The Root returns the address of the .com TLD nameserver.
3. Resolver → TLD. The Resolver queries the .com TLD server, which returns the address of the amazon.com authoritative nameserver.
4. Resolver → Authoritative. The Resolver queries the authoritative nameserver, which returns the actual IP address of amazon.com.
5. Resolver caches → Browser. The Resolver caches the result and hands the IP to your browser, which then requests the site directly from that IP.

Caching keeps this fast. The whole DNS infrastructure is a distributed system living in large ISP data centers, with server clusters tuned to answer queries in milliseconds. Results are cached along the way — and your own device caches DNS records too, with a TTL (Time To Live). Modern browsers do this automatically to skip the lookup on repeat visits.

A service the size of amazon.com runs across many data centers worldwide, and DNS load balancing spreads traffic across them right at the authoritative nameserver. Instead of returning a single IP, the authoritative server returns a list of IPs for the domain — and with each query it rotates the order of that list in round-robin fashion.

The client tries the first IP on the list; the reason it gets a whole list is so it can fall back to the next IP if the first doesn't respond in time. The next client's query gets the list re-ordered with a different IP on top, and so on. Note that the IP a client hits may not be an application server directly — it's often another load balancer at the data-center level that manages that site's server clusters.

There are three modes of load balancing overall: DNS (covered above), hardware-based, and software-based. The latter two are the common ways to balance traffic on large-scale services.

The trade-off in one line: hardware wins on peak performance, software wins on cost, flexibility, and how much smarter its routing decisions can be — which brings us to the algorithms.

DNS balancers only know round robin. Software balancers add several smarter algorithms.

Round robin & weighted round robin

Round robin hands out server IPs sequentially, ignoring load, CPU, or anything else. Weighted round robin fixes the blind spot: each server gets a weight based on its compute and traffic-handling capacity, and traffic is distributed in proportion to those weights. More traffic flows to the beefier machines — useful when a service spans multiple data centers of different sizes, so the larger ones absorb more.

Least connections

Routes the next request to the machine with the fewest open connections. Two flavors:

• Naive: assumes every request costs the same resources — but the machine with the fewest connections might already be pinned at high CPU, so sending more there is a bad bet.
• Resource-aware: also weighs CPU utilization and request-processing time. The ideal target has the shortest processing time, lowest CPU use, and fewest open connections.

Least connections shines when connections stay open a long time — like persistent connections in a gaming application. [+]

Random

Routes traffic to servers at random. The balancer may also group servers that look similar (by existing load, processing time, etc.) and then randomly pick among that similar set.

Hash

Hashes the source IP and/or the request URL to decide the target — and the payoff is stickiness:

Hashing the source IP guarantees a given client always lands on the same server. That server already holds the client's data in local memory, so it skips fetching session data from the cluster's shared session store — lower latency — and lets the client re-establish with the same server if the connection drops. Hashing the URL guarantees a request always hits the cache that already holds its data, avoiding cache misses and removing the need to duplicate data across every cache.

The section cites two external links. Both verified live today.