Notefoundation

API Gateway & Load Balancer

The two components every client request hits first — what the gateway does (auth, rate limiting, routing), how a load balancer spreads traffic, and the one idea interviewers probe: plain hashing vs consistent hashing and its replica fix.

35 min read·verified Content review 2026·reviewed 2026-08-30

Assumes you know

Networking Essentials

After this you should be able to

Explain the core idea in API Gateway & Load Balancer
Apply API Gateway & Load Balancer in a representative design

API GatewayRate LimitingConsistent Hashing

System Design · High-Level Design

The two components every client request hits first. A ground-up walkthrough of what each does, why an interviewer wants to hear about them, and the one idea — consistent hashing — that separates a strong answer from an average one.

Items marked [+] were added or expanded beyond the original session. Callouts are tagged ★ trick · ⚠ pitfall · 💡 tip.

0:00 · the whiteboard moment that grades the interview

“Design Instagram.” You uncap the marker, draw a box for the phone, a box for the database, and join them with a line.

It works on paper. It also tells the interviewer you've never run anything at scale — because a real client never touches the backend directly. Between them sits a box every strong candidate draws first, and then gets quizzed on for ten minutes: why is it there?

That box is two jobs wearing one label. The API Gateway is the security desk for the whole backend — it checks who you are, throttles abuse, and routes you to the right office. The Load Balancer is the traffic cop standing behind it, spreading requests across identical servers so none of them buckles. Learn these two cold and you've aced the part of the interview that happens before you've drawn anything interesting.

I. The First 15 Minutes — Interview Framework#

1. The opening sequence#

When asked to “design X”, run a fixed sequence before you draw a single box. This is High-Level Design (HLD / architectural design), a different round from Low-Level / object-oriented design — if it's unclear which you're in, ask. The opening always follows the same five beats:

ClarifyScopeFunctional reqsNon-functional (SCARL)Traffic

Clarify — if you don't know the app, ask what it does. The interviewer will not prompt you to ask; they're watching whether you do.
Establish scope — you can't build everything, so fix the boundaries early.
Functional requirements — the features. Pitch the ones you're most confident in, in ranked order (1, 2, 3…), because you'll likely implement them in that order.
Non-functional requirements — remembered as “Don't Be SCARL”: Scalability, Consistency, Availability, Reliability, Latency. [+] These barely change between HLD problems, so treat them as pre-memorized.
Traffic estimation — DAU / MAU. This is what drives scalability: high traffic → you need a load balancer and multiple servers.

Analogy · recall The opening is a pre-flight checklist, not improvisation. A pilot doesn't decide which gauges to read once the plane is moving; they run the same list every time so the interesting part — the flight — has their full attention. Same here: fixed checklist first, so the hard design thinking gets your whole brain.

Why it matters: don't add a feature you can't implement — if you list it, you own building it. Only pitch functionality you actually understand and let the interviewer explicitly request anything beyond that. [+] Naming SCARL and a rough DAU number in the first few minutes signals seniority before you've drawn a single component.

2. Why the first 15 minutes matter — and how to “leave hints”#

Acing the opening buys you thinking time later and sets the first impression that colors the whole interview. Because the non-functional requirements are essentially the same every time, you should never be inventing them live.

⭐ TRICK — “leave hints” for your interviewer. System design is an open discussion with no single right answer (unlike DSA). You can steer it by deliberately mentioning a component or trade-off, which invites the interviewer to ask the exact follow-up you've prepared. We'll cash this trick in at the end, in the consistent-hashing section — watch for it.

II. API Gateway#

1. What it is#

Many clients, many services — one guarded entry point funnels everything

A central component that is the single entry point to all your microservices. It works as a reverse proxy, sitting between clients (web, mobile, third-party) and backend services, and handles authentication, authorization, routing, rate limiting, and request/response transformation.

Analogy · recall Think of it as the security desk in an office lobby. Visitors don't wander into individual offices; they pass one desk that checks ID, decides where they're allowed, and points them down the right corridor. The offices (backend services) get to focus on their actual work instead of vetting every stranger at their own door.

2. Single / unified entry point#

Every client request must go through the gateway. Clients should never hit backend services directly — there must always be a component separating them. That's what makes the gateway a “unified entry point.”

⚠ Pitfall — single point of failure: one gateway for the whole backend means if it dies, the entire system collapses. Fix: run multiple replicas of the gateway, distributed across different data centers, so a regional outage still leaves a live gateway elsewhere. [+] This is a near-guaranteed follow-up — volunteer it before they ask.

3. Authentication & authorization#

The gateway is a filter: only clean, permitted requests reach the backend world

Centralize auth at the gateway so backend services don't each re-implement it. The gateway validates incoming credentials (JWT token, API key, or row/role access token). Valid → forward to backend. Invalid → return 401 Unauthorized (authentication fails — we don't know who you are) or 403 Forbidden (we know you, but you're not permitted). For authorization it reads the user's token, extracts claims (user ID, role e.g. admin vs user, permissions), and checks whether that user may hit the requested API.

💡 Tip — why this is the “why”: if every backend endpoint did its own auth on top of its business logic, each adds latency, and with many services that latency stacks before the client gets a response. Centralizing it (1) cuts latency, and (2) keeps malicious/unauthorized requests out of the “backend world” entirely — better security and compliance.

⚠ Pitfall — two auth layers, both exist: the gateway authenticates the client (is this caller allowed into the backend at all?). Inside the backend, a service still does consumer-level authorization (is this specific consumer on this API's allow-list? if 10 consumers are listed, the 11th is rejected). Framework security like Spring Security governs that internal allow-list; the gateway governs entry. Don't conflate them. [+]

💡 Tip — how deep to go depends on the company: for a normal product company (Amazon, Uber) just naming auth and its purpose is enough. Interviewing for a security / auth-specialized team? Expect them to dive deep — prepare more there.

4. Request / response transformation#

The gateway speaks both dialects — backends can change shape without breaking the client

The gateway converts between formats and API versions so backends can evolve without breaking clients. It decouples the internal data representation from the external API. Example: a client sends XML, but every backend API expects JSON — the gateway converts XML→JSON before forwarding, then JSON→XML on the way back. Each API publishes an API contract (header + request body + response shape) so the gateway knows the expected format. [+] Services communicate via APIs; a request carries a header and a body of attributes, and the server returns a response body after running its business logic.

5. Rate limiting & throttling#

Leaky bucket — water in at any rate, out at a fixed drip; the excess spills

Caps how many requests a client can make, protecting backends from overload. If an API handles 100 TPS and requests keep flooding past that, CPU climbs toward 100% and the system can go down — that hurts availability. The gateway absorbs the flood but only forwards a controlled rate, dropping the excess.

Analogy · recall A leaky bucket: water (requests) pours in at any rate, but drains from a small hole at the bottom at a constant rate. No matter how fast it comes in, the outflow to your backend stays steady; overflow spills over the rim and is dropped. A retry variant instead holds the excess and replays it after 5–10s — conceptually like parking requests in a message queue for async processing.

⭐ Trick — spot the attacker by behavior, not identity: you can't tell a malicious user from a normal one up front. But a human can't fire 1,000 requests a minute — that anomalous rate is the tell. Rate limiting stops the abuse on the very first burst, so you never give the attacker even one chance to topple the system.

⚠ Pitfall — it isn't only for attackers: even legitimate traffic can exceed capacity — if an API handles 100 TPS and several genuine clients together send 120 in a second, the limiter still protects the backend. (Genuine spikes — New Year's Eve, a celebrity post — are handled by scaling, and hot pages by caching; both are separate topics.)

6. Caching#

The gateway stores frequently requested responses and serves repeats itself. When the same request recurs, it returns the stored response instead of calling the backend. Result: less load on backend services and faster response time, because the round-trip to the backend is skipped entirely.

7. Protocol translation#

Bridges different communication protocols between client and backend. The client↔gateway link might be HTTP while the backend services talk WebSocket internally; the gateway translates between them — same idea as request/response transformation, but for the protocol layer.

8. Service discovery / routing#

The gateway knows who owns what — endpoint 15 lives in service 2, so that's where it goes

The gateway knows which backend service owns which endpoints, and routes each request there. If service 1 owns APIs 1–10, service 2 owns 11–20, service 3 owns 21–30, then a request for endpoint 15 (after passing auth) goes to service 2. Concretely it examines the request's path, method, and headers, matches them against configured routing rules, and forwards to the right service.

💡 Tip — “routing” and “service discovery” are the same thing here: these are effectively identical in this context — both identify the satisfying service and forward the request.

⭐ Trick — when to split discovery into its own component: the gateway already does many jobs. With hundreds of services and heavy traffic, doing discovery inside the gateway adds load — offload it then. Analogy: if you already juggle 15 projects and 5 more arrive, the manager hires someone for the new 5 rather than crushing you. Most real microservice setups keep discovery separate — but the gateway can do it; there's no single right answer.

9. When to use an API Gateway#

You're on a microservices architecture.
You support multiple clients (web, mobile, third-party APIs).
You need cross-cutting features: auth, plus logging (record each request/response) and monitoring (e.g. a Datadog dashboard to read traffic and decide when to scale — the gateway faces the outside world, so it's the natural place to watch traffic).
Scalability matters.
The system changes often (services added/removed, APIs updated) — the gateway's routing keeps up as endpoints come and go.

III. Load Balancer#

1. What it is#

One entry point, work spread across healthy servers — the dead one (S4) is pulled from the pool

Distributes incoming traffic across multiple servers so no single server is overloaded. If one server handles 50 requests and 150 arrive, it can't cope. Following horizontal scaling you add more identical servers — and the load balancer decides which request goes to which server.

Goals (its “why”): High Availability (one server dies, others serve), Better Performance (no server is maxed, so CPU stays low and responses are quick), Reliability, and Scalability.

2. Components#

Listener — receives incoming traffic; the entry point.
Backend servers — the actual servers processing requests; the LB picks which one.
Health checks — continuously monitor servers and remove unhealthy ones from the pool.
Distribution algorithm — decides which server gets each request (algorithms below).

End to end: user sends request → reaches the LB → LB forwards to a backend server per the chosen algorithm → server responds. Alongside this it maintains session persistence, runs health checks, and supports adding/removing servers without users noticing (horizontal scaling).

3. Session persistence (sticky sessions) & the TCP cost#

Three round-trips before any data flows — so pin the client and reuse the channel

Keep a given client's requests going to the same server for the duration of a session. Useful for stateful applications. The reason is the TCP three-way handshake: SYN (client→server) → SYN-ACK (server→client) → ACK (client→server) establishes the connection before any request/response flows.

⚠ Pitfall — the handshake is costly: re-establishing TCP on every request adds latency. So once a connection is up, the LB keeps the client pinned to the same server for ~10–15 minutes and reuses the channel — avoiding repeated handshakes and lowering overall latency.

⚠ Pitfall — health check ≠ handshake: the three-way handshake establishes a TCP connection between client and server. A health check is the load balancer probing servers to learn which are alive and belong in the pool. Different mechanisms entirely. [+]

4. Real-world uses#

Web apps — spread HTTP requests across web servers.
Database servers — balance read/write requests across DB servers.
Microservices — route to the appropriate service (this overlaps with the gateway's service discovery; either component can do it).
CDNs — send a user to the nearest server. A user in Bangalore is served by the Bangalore server, not Itanagar, because physical distance adds latency. Geographically distributed servers also give regional resilience.

💡 Tip — both components can do both jobs: the gateway can load balance; the LB can route/discover. We keep the LB separate here precisely because the gateway is already busy — overloading it risks taking it down.

IV. Health Checks#

Five ways to verify a server is healthy#

Only healthy servers should receive traffic; an unhealthy one is removed from the pool. The load balancer keeps probing — respond and you're in; fail to respond in time and you're pulled out.

Type	How it checks	Healthy when…
Ping	Sends a ping request	Server responds
TCP	Tries to open a TCP connection	Connection succeeds
HTTP / HTTPS	Sends an HTTP request to an endpoint	Expected response (usually 200 OK)
Application-specific	Calls an app endpoint like `/health`	Endpoint reports OK
Response-time monitoring	Watches server response time	Below threshold (over → unhealthy)

V. Load Balancing Algorithms#

1. The six, at a glance#

Algorithm	Rule	Pro	Con
Round Robin	R1→S1, R2→S2, R3→S3, R4→S4, R5→S1… (sequential, loops)	Simple; fine for equal-capacity servers	Ignores server load & capacity
Least Connections	Send to the server with fewest active connections (S4 with 5 beats S1 with 100)	Better distribution; handles varying workloads	Needs continuous monitoring
Least Response Time	Send to the fastest-responding server	Improves responsiveness	Needs continuous monitoring
IP Hash	Hash client IP → server (below)	Maintains session persistence	Brittle when server count changes
Random	Pick a server at random per request	Very simple	May not distribute evenly
Consistent Hashing	Servers + requests on a hash ring; serve clockwise (below)	Minimal redistribution on change; standard in distributed systems	More complex to implement

2. IP Hash (plain hashing) — and why it's brittle#

The divisor is the server count — change N and the whole map moves

Hash the client's IP to pick a server, giving the same client the same server every time. A hash function turns the IP into a number, then you take number mod (server count). Example: hash → 261, with 4 servers → 261 mod 4 → a fixed server; that client sticks there (so session persistence holds, and after the first TCP handshake the connection is reused).

⚠ Pitfall — the modulus is the server count: drop from 4 servers to 3 and you now compute mod 3, so almost every client's mapping changes — forcing mass re-orientation and re-established sessions. The same happens on a server crash (one fewer healthy server) or a scale-up. This is the core weakness consistent hashing fixes.

VI. Consistent Hashing — the key idea#

Servers sit on a circle; each request walks clockwise to the first server — change one, move only its share

1. How it works#

Place servers on a ring; for each request, walk clockwise and assign it to the first server you hit. There is no mod-by-server-count step — that's the whole point.

⚠ Pitfall — three names people blur: plain hashing uses IP mod N servers; consistent hashing uses the clockwise ring; and there is no such thing as “persistent hashing.” It's also not Round Robin — that cycles R1→S1, R2→S2 by position; consistent hashing assigns by “first server clockwise from the request's spot.” Keep all three distinct. [+]

2. Why it beats plain hashing#

When a server is added or removed, only minimal requests re-map. Remove S2 and only R1, R2 (its requests) move to the next clockwise server (S3); R5–R8 aren't touched. Add a server between two requests and only the one now closest clockwise re-routes. Contrast plain hashing, where any change to server count remaps everyone.

3. The cascading-failure problem & the replica fix#

Scatter virtual copies of each server around the ring, so one failure spills onto several neighbors, not one

⚠ Pitfall — removal can topple the next server: if S2 (handling a big share) dies, all of it shifts to S3, which is already busy. S3 falls over, its load shifts to S4, and on it goes — a cascading failure that can take the whole system down. (Even the “80% capacity buffer” used in practice isn't enough for a sudden viral burst.)

⭐ Trick — scatter replicas (virtual nodes): instead of S1, S2, S3, S4 sitting once each, place a few instances of each (S1-a, S1-b, S2-a, …) at many points on the circle. A replica is just a Xerox copy of the server — same endpoints, an extra instance. Now when one server dies, its load splits among several different next-clockwise servers instead of all landing on one — distribution stays even and nobody gets crushed. Add only 1–3 replicas per server, not hundreds — just enough to spread load.

Analogy · recall Plain hashing is assigned seats by row number — renumber the rows (add/remove a server) and everyone moves. Consistent hashing is a clock face: you stand where you stand and walk clockwise to the next open service desk. Close one desk and only the people who were heading to it walk a little further; everyone else doesn't budge.

4. The “leave a hint” interview play#

⭐ Trick — script the answer to bait the question you want: when asked about the load balancer, say: “I'll use a load balancer; here are its advantages, and my algorithm is hashing — hash the client IP, mod the server count, route there.” Add the TCP angle: “This keeps session persistence, so I avoid repeating the three-way handshake — lower latency.” Then plant the hint: “The one downside is that on scale up/down the server count changes and all requests re-map — so I'd move to consistent hashing.” You've now invited the interviewer to ask “how does consistent hashing fix that?” — exactly what you're ready to explain. The likely next probe is the cascading-failure case (answer: replicas).

💡 Tip — the ring is a teaching model, not a literal build: if pushed on “which of 100 requests goes where initially,” the honest answer is: distribute as uniformly as possible (random start or by IP). In a real system there's no literal circle of data — it's a concept showing how minimal re-mapping is achieved. Don't over-engineer the “real world” version in the interview.

VII. Interview Q&A — drill these aloud#

Read the story above first, then close it and answer these out loud — try before you expand. They're the real follow-ups this material invites.

Q1 — Why can't the client just call the backend services directly?

Because every service would then re-implement auth, rate limiting, logging, and routing on top of its business logic — stacking latency and scattering security. A gateway centralizes those cross-cutting concerns at one guarded entry point, so services stay focused and only clean, authorized traffic ever reaches them.

Q2 — The gateway is a single point of failure. How do you fix that?

Run multiple replicas of the gateway, distributed across different data centers. If one gateway (or a whole region) goes down, another live instance still serves traffic. Volunteer this before they ask — it signals you think about availability.

Q3 — What's the difference between 401 and 403?

401 Unauthorized = authentication failed; we don't know who you are (bad/missing token). 403 Forbidden = we authenticated you, but you lack permission for this resource. First answers “who are you?”, the second “are you allowed?”

Q4 — A malicious user floods your API. How does rate limiting help, and how do you tell them from a genuine spike?

The gateway caps the forward rate (leaky bucket: absorb bursts, drain at a constant rate, drop overflow), protecting availability. You distinguish abuse by behavior, not identity — no human sends 1,000 requests a minute, so the anomalous rate is the tell. Genuine mass spikes (New Year's Eve, a celebrity post) are handled separately by scaling and caching.

Q5 — Why use session persistence / sticky sessions?

To avoid repeating the costly TCP three-way handshake (SYN → SYN-ACK → ACK) on every request. Once a connection is established the LB pins the client to the same server for ~10–15 minutes and reuses the channel, lowering latency. It's also needed for stateful apps where the server holds session state.

Q6 — Round robin vs least connections — when does round robin fall short?

Round robin cycles requests sequentially and assumes every server has equal capacity and load — it ignores how busy each one actually is. When servers differ or some requests are heavy, least connections (route to the server with the fewest active connections) distributes far better, at the cost of continuously monitoring connection counts.

Q7 — What exactly breaks with plain IP-hash load balancing when a server is added or removed?

The mapping is hash(IP) mod N, where N is the server count. Change N (scale up/down or a crash) and the modulus changes, so almost every client remaps to a different server — every session has to be re-established at once. That mass reshuffle is exactly what consistent hashing eliminates.

Q8 — How does consistent hashing reduce remapping, and what's its catch?

Servers and requests sit on a ring; each request is served by the first server clockwise — no mod-by-N. Add or remove a server and only the requests in that arc move; the rest stay put. The catch is cascading failure: a dead server dumps its whole load on the next, which can topple it. The fix is replicas / virtual nodes — scatter a few copies of each server around the ring so a failure's load splits across many neighbors instead of one.

Q9 — Gateway vs load balancer — they overlap on routing and balancing. Why keep them separate?

Either component can route and can balance — there's no single right answer. We separate them because the gateway already carries many jobs (auth, transformation, rate limiting, caching, discovery); piling load balancing on top risks overloading and taking it down. Keeping the LB separate isolates that critical, high-volume role.

Recall Card copy by hand#

1. HLD opening sequence#

Clarify → Scope → Functional → Non-functional → Traffic • don't know app? → ask clarifying Qs (they won't prompt you) • functional reqs → pitch ranked (1,2,3); only what you can build • non-functional = "Don't Be SCARL" = Scalability · Consistency · Availability · Reliability · Latency • Traffic (DAU/MAU) → drives Scalability → need LB + more servers? ★ ace first 15 min → buys thinking time; first impression is critical ★ "leave hints" → steer interviewer to the Q you've prepared [+] HLD ≠ LLD — separate rounds; ask which one

2. API Gateway#

= single entry pt to microservices; REVERSE PROXY; wall (outside | backend) 1. Unified entry → all client reqs go through it 2. Auth/Authz → 401 unauth / 403 forbidden; reads JWT claims (userID,role,perms) 3. Req/Resp xform → e.g. XML ↔ JSON; backends evolve w/o breaking client 4. Rate limit → cap reqs; protect from overload 5. Caching → store hot responses; less backend load, faster 6. Protocol xlate → HTTP ↔ WebSocket 7. Discovery/route→ knows which svc owns which endpoints → routes ⚠ single point of failure → replicate gateway across data centers ⚠ gateway auth (entry) ≠ in-service auth (consumer allow-list, e.g. Spring Security) ★ centralize auth → backends skip it → less stacked latency + blocks bad reqs early ★ heavy load + 100s of services → split discovery into its own component Use when: microservices · multi-client · auth/logging/monitoring · scales · changes often

3. Rate limiting — leaky bucket#

water in @ ANY rate → drains @ CONSTANT rate from bottom hole; overflow dropped ★ detect attacker by BEHAVIOR not identity → human can't send 1000 req/min = anomaly ⚠ not just attackers — genuine clients can exceed capacity too + retry variant: hold excess, replay 5-10s ~ message-queue parking

4. Load Balancer#

= spreads traffic across servers; nobody overloaded Goals: High Availability · Performance · Reliability · Scalability Parts: Listener (entry) · Backend servers · Health checks · Distribution algo Flow: user → LB → server (per algo) → response Session persistence = pin client to same server (~10-15 min); stateful apps WHY: TCP 3-way handshake (SYN → SYN-ACK → ACK) is COSTLY → reuse conn → ↓latency ⚠ health check (LB probes who's alive) ≠ handshake (sets up TCP conn) Real uses: web · DB read/write · microservices · CDN (nearest server: Bangalore not Itanagar) ⚠ LB can route & gateway can balance — keep separate so gateway isn't overloaded

5. Health checks#

Ping · TCP · HTTP/HTTPS (200 OK) · App-specific (/health) · Response-time (over threshold → unhealthy) → only healthy servers get traffic; unhealthy pulled from pool

6. LB algorithms#

Round Robin → R1→S1,R2→S2… loops | con: ignores load/capacity Least Conns → fewest active conns | con: needs monitoring Least Resp Time → fastest server | con: needs monitoring IP Hash → hash(IP) mod N → server | con: remaps all on N change Random → random server | con: uneven Consistent Hash → ring, serve clockwise | con: complex; cascade risk

7. Hashing vs consistent hashing#

Plain: hash(clientIP) → num → num MOD (server count) → server; keeps session ⚠ scale up/down OR crash changes N → mod changes → ALL reqs remap Consistent: servers on a RING; request walks CLOCKWISE → first server. NO mod-by-N. ⚠ consistent hashing ≠ plain hashing ≠ round robin; no "persistent hashing" ✓ remove S2 → only R1,R2 → S3; R5-R8 untouched. Add server → only nearest req moves. ⚠ CASCADE: dead server's load → next → it dies → next → whole system down ★ FIX = replicas (virtual nodes): scatter S1a,S1b,S2a… around ring → load splits evenly (1-3 replicas/server, not 100; replica = Xerox of server) ★ INTERVIEW: pitch hashing → mention session/TCP → name remap flaw → hint consistent hashing

VIII. Key Takeaways#

Open every HLD problem the same way: Clarify → Scope → Functional → Non-functional (SCARL) → Traffic. It buys time and a strong first impression.
API Gateway = one guarded door to the backend: auth/authz, request/response transformation, rate limiting, caching, protocol translation, routing/discovery, logging/monitoring. Centralizing these takes load off backend services.
The gateway is a single point of failure → replicate it across data centers.
Load Balancer = traffic cop across identical servers; goals are availability, performance, reliability, scalability. It runs health checks and session persistence (to dodge repeated TCP handshakes).
Know the six algorithms, but the differentiator is plain vs consistent hashing: plain remaps everyone on any server-count change; consistent (ring + clockwise) remaps only the affected requests.
Consistent hashing's catch is cascading failure when one server's load lands on the next — fixed with replicas / virtual nodes spread around the ring.
Interview meta-skill: leave hints to steer the interviewer toward the follow-ups you've prepared.

API Gatewayreverse proxySCARL 401 / 403rate limitingleaky bucket load balancerhealth checkssession persistence TCP handshakeround robinleast connections IP hashconsistent hashingvirtual nodes cascading failureleave a hint