Stop 17 · CI/CD & DeploymentTopics · 9/9

Load Balancing Basics

In short

One server can only take so many requests. Add a second and someone has to decide who handles each request. That someone is a load balancer, and it's simpler than it sounds.

Distributing incoming requests across multiple server instances so no single instance is overwhelmed — necessary once one server can't handle your traffic alone.

The purple blob hands out tokens and waves customers to three open bank counters.

Think of it like this

The bank token machine

At the bank, you take a token and the display sends you to whichever counter is free. If counter 3's clerk goes for lunch, tokens stop going there. You don't care which clerk you get, only that the queue keeps moving.

Watch it happen

Round robin with a health check

1/5Request 1 arrives. Round robin sends it to S1, the first server in the list.

The big ideas

What you need to know

Round robin

Hand requests to servers in turn. Simple and fair when every request costs about the same.

Least connections

Send the next request to the server with the fewest open connections. Better when some requests are slow.

Health checks

The balancer keeps checking each server and stops sending traffic to any that fail.

Layer 4 vs layer 7

L4 balancers forward TCP connections without reading them. L7 balancers read HTTP, so they can route by path or header.

See it in code

Walk through the code

nginx

1upstream api {2  least_conn;                    # pick the least busy server3  server 10.0.0.11:3000;4  server 10.0.0.12:3000;5  server 10.0.0.13:3000 max_fails=3 fail_timeout=30s;6  server 10.0.0.14:3000 backup;  # only used if the others fail7}8 9server {10  listen 80;11  location / {12    proxy_pass http://api;13  }14}

1/4Without this line Nginx uses round robin. least_conn picks the server with fewest active connections.

Open-source Nginx marks a server down after failed requests (passive checks). Managed balancers like AWS ALB also poll a health URL actively.

Save yourself the pain

Common mistakes

Oops

Keeping uploaded files on the local disk of one server.

Do this instead

The next request may land on a server that doesn't have the file. Put files in S3 or shared storage.

Oops

Adding a load balancer in front of one database and calling it highly available.

Do this instead

The balancer spreads app load. If the single database dies, everything still dies. Plan for that too.

Your mission

Balance three local servers

~30 min

Quick check

Some requests take 5 seconds (reports) and most take 50 ms. Which algorithm spreads the load better?

Go deeper

1Configure Nginx to load-balance between two local instances of the same app.
2Learn the difference between round-robin, least-connections, and IP-hash balancing strategies.
3Understand why load balancing requires your app to be stateless (or use sticky sessions) to work correctly.