Load Balancers and API Gateways

A load balancer decides which backend receives work. An API gateway decides which requests may enter an API and how they reach its services. Both often operate as reverse proxies: clients connect to the intermediary, which contacts servers on their behalf.

These are responsibilities that can coexist in one product. Putting a proxy before a server pool gives clients a stable entry point while replicas start, fail, or leave. Adding route and access policies gives an expanding API a shared enforcement point. Neither responsibility removes the need to design the services behind it.

Choose a unit of work before a balancing algorithm

A layer 4 balancer selects a destination using transport information, such as addresses and ports. A TCP connection stays attached to its selected backend. A layer 7 HTTP proxy understands individual requests and can route /orders and /reports to different pools. With TLS pass-through, encrypted HTTP remains opaque to the intermediary; terminating TLS makes HTTP routing possible. The proxy can establish a separate encrypted connection upstream.

Equal connections do not imply equal work. An HTTP/2 connection can carry many concurrent streams. One upgraded WebSocket can remain open for hours. At layer 7, equal request counts still hide differences between a fast lookup and an expensive report.

Round robin spreads selections around the pool. Weights let a larger replica receive a larger share. Least-connections policies prefer fewer active connections, which can help with uneven durations but imperfectly represents CPU work or multiplexed requests. Some proxies expose outstanding-request policies instead. Hashing a tenant or session key improves locality, but a large tenant can make one destination hot.

Session affinity adds a constraint: related requests return to the same replica. It can preserve an in-memory session, but that session disappears if the replica dies, and a sticky client can keep a draining replica busy. Keeping durable session state outside the handler process allows more freedom to rebalance. The right algorithm follows the workload and its failure behavior; it does not create spare capacity.

AWS products illustrate the overlap:

Authentication is therefore not an exclusive definition of an API gateway. API keys identify API consumers for usage management; they are not a substitute for authenticating users or checking resource ownership.

Watch two backends behind one proxy

Download the runnable fixture. Prerequisites are Python 3.11+ on the host and a working Docker daemon. The runner uses pinned official Python and HAProxy image digests, starts two disposable containers, publishes no ports, and removes its containers and temporary volume afterward. Its first run needs access to pull the images. It changes no application dependencies.

From the extracted directory, run:

python3 docker_checks.py
# Runs configuration validation, real proxy checks, then four policy tests.

The backends listen on loopback ports 9001 and 9002 inside a shared container network namespace. They identify themselves in JSON. This is the request-handling core of backend.py; the complete file supplies the handler factory, imports, and server startup:

def do_GET(self):
    status = 200
    if self.path == "/ready":
        status = 503 if (state / f"unready-{name}").exists() else 200
    elif self.path == "/slow":
        (state / f"started-{name}").touch()
        until = time.monotonic() + 3
        while not (state / f"release-{name}").exists():
            if time.monotonic() >= until:
                status = 503
                break
            time.sleep(0.01)
    elif self.path != "/":
        status = 404
    body = json.dumps({
        "backend": name,
        "request_id": self.headers.get("X-Request-ID", str(uuid4())),
        "forwarded_for": self.headers.get("X-Forwarded-For"),
        "forwarded_proto": self.headers.get("X-Forwarded-Proto"),
    }).encode()
    self.send_response(status)
    self.send_header("Content-Type", "application/json")
    self.send_header("Content-Length", str(len(body)))
    self.end_headers()
    self.wfile.write(body)

ThreadingHTTPServer matters here: a slow request must not prevent another connection from answering readiness probes. A marker makes readiness fail while the process stays alive. A separate admission marker lets the test prove that a slow request reached A before draining A; a release marker then completes it without relying on a lucky sleep. Without explicit release, the three-second watchdog returns 503, so a successful response proves release occurred after drain.

This is the complete haproxy.cfg. The short check interval is for the lab, not a production sizing recommendation:

global
  stats socket /state/admin.sock mode 600 level admin

defaults
  mode http
  timeout connect 1s
  timeout client 10s
  timeout server 5s
  retries 0

frontend public
  bind 127.0.0.1:8080
  http-request set-header X-Forwarded-For %[src]
  http-request set-header X-Forwarded-Proto http
  default_backend app

backend app
  balance roundrobin
  option httpchk GET /ready
  http-check expect status 200
  server a 127.0.0.1:9001 check inter 200ms fall 2 rise 2
  server b 127.0.0.1:9002 check inter 200ms fall 2 rise 2

Before starting the proxy, the runner executes haproxy -c -f /lab/haproxy.cfg. With both replicas ready, twenty sequential requests on fresh connections returned ten responses from each in the tested configuration. Changing A's weight to three produced thirty A responses and ten B responses across forty requests. That is a controlled observation, not a promise of alternating responses under concurrency, affinity, or failure.

The header test supplies a forged X-Forwarded-For: 203.0.113.99. The backend instead sees 127.0.0.1, because this edge proxy replaces the client's assertion. It likewise replaces the forwarded scheme with http, which accurately describes this intentionally unencrypted lab. A request ID survives forwarding for correlation; it grants no authority.

Removing a replica is a sequence

Readiness asks whether a backend should receive new work. Process survival asks only whether it still exists. A shallow port check can pass while the application cannot serve requests; an overbroad readiness dependency can remove every replica during one shared dependency failure. Design the probe around what that service can actually serve.

Here, two consecutive failed HTTP checks remove a replica; two successes restore it. Detection includes the probe schedule, response duration, and thresholds. It is not instantaneous, and removal transfers load to the survivors. The remaining replicas need enough capacity for that transfer.

The drain check uses the fixture's Unix-socket runtime() helper. Temporarily draining B forces admission on A. After the admission marker appears, B returns and A leaves the selection pool:

runtime("set server app/b state drain")
with ThreadPoolExecutor(max_workers=1) as pool:
    slow = pool.submit(request, "/slow")
    eventually(lambda: (STATE / "started-a").exists(), "A admitted work")
    runtime("set server app/b state ready")
    runtime("set server app/a state drain")
    assert statuses()["a"] == "DRAIN"
    assert_backends({"b"})
    assert not slow.done(), "slow request completed before explicit release"
    (STATE / "release-a").touch()
    status, reply = slow.result()
    assert status == 200 and reply["backend"] == "a"
runtime("set server app/a state ready")
# A's admitted request completes; fresh requests use B.

There is no persistence configured here. Draining stops ordinary new selections; it does not migrate existing requests or streams. Production shutdown usually removes readiness or starts draining, waits for admitted work within a bounded budget, then stops the process. Long-lived streams need a reconnection strategy when that budget expires.

The runner separately fails A's readiness and polls actual proxy status before asserting that only B serves requests. It also holds a stale health decision by briefly disabling A's checks, kills A, and sends a request with B drained. That request returns 503 with retries disabled. This deterministically exposes the interval in which the proxy's last health decision can disagree with reality. Restoring checks removes A; making B unready as well produces HAProxy's no-available-backend 503. A balancer cannot manufacture a healthy destination.

Enforce admission without replacing service authorization

Gateway policy starts with authenticated claims. A production token verifier must check the signature with trusted keys and permitted algorithms, then issuer, audience, and expiry. A readable JWT payload is not verified identity. The fixture injects a token-to-claims lookup as a test double; it implements no cryptography and accepts no real credentials.

Its policy model uses one known route, a per-principal limit of two admissions per ten-second window, and a maximum total budget of 500 ms:

claims = verify(token)  # Trusted verifier in production; test lookup in this lab.
now = clock()
if claims is None or claims.issuer != "trusted-issuer" or claims.audience != "orders-api" or now >= claims.expires:
    raise Rejected(401, "invalid token")
if path != "/orders/42":
    raise Rejected(404, "unknown route")
if "orders:read" not in claims.scopes:
    raise Rejected(403, "missing route scope")
key = (claims.tenant, claims.subject, int(now // 10))
if used[key] >= 2:
    raise Rejected(429, "rate limit", max(1, math.ceil(10 - now % 10)))
used[key] += 1
deadline = now + min(500, max(0, requested_ms)) / 1000

# At the service, after admission, using the model's shared clock:
if clock() >= deadline:
    raise Rejected(504, "deadline exhausted before work")
if claims.tenant != order_tenant:
    raise Rejected(404, "order not found")

Tests reject an unknown token, wrong audience, wrong issuer, and a token exactly at expiry. Missing route scope returns 403. A valid tenant A identity cannot read tenant B's order. The third admission gets 429 with a retry delay; the next window admits again. A huge requested budget is clamped, and work at the exact deadline is rejected. Fixed windows can burst around their boundary; this local counter does not enforce a global quota across gateway replicas.

Trusted identity must travel over a protected service boundary. Strip client-supplied identity headers before setting verified ones, trust forwarding headers only from configured proxies, and restrict direct backend access. The load-balancing fixture tests forwarding-header replacement; it is not an authenticated gateway.

The configuration's connection and inactivity timeouts are not an end-to-end deadline. A service can keep producing bytes and extend its total elapsed time. Propagate a bounded remaining budget across hops, clamp client requests to server limits, and check whether useful work can still finish. A timeout response alone does not cancel backend work or roll back a committed write. The fixture's deadline check models rejection before work; it does not demonstrate cancellation.

Assign retries to a deliberate owner with a bounded attempt and time budget. Three layers each allowing three total attempts can produce twenty-seven backend attempts. Retrying writes also needs an idempotency strategy, developed in Queues and Retries. Finally, provision the gateway itself: its connections, queues, TLS work, token verification, and shared policy dependencies can become the bottleneck that every otherwise healthy service shares.