Every service of any size has something in front of it. Understanding the layer at which it operates, what it can therefore see, and what it does when things go wrong is one of the most practically useful areas in infrastructure — and health checks in particular cause more outages than the failures they are meant to protect against.

A layer-4 balancer forwards packets based on addresses and ports. It cannot see a URL, a header or a cookie, because it never decrypts anything. It is extremely fast, it preserves the client IP naturally, and it can handle millions of connections per second on modest hardware.

A layer-7 balancer terminates the connection, reads the request, and makes a new connection to a backend. That gives it everything — path-based routing, header-based canaries, retries, rate limiting, request rewriting, buffering of slow clients — at the cost of latency, CPU and, importantly, becoming a component that holds state and can itself fail.

The rule of thumb worth stating: use L4 when you need throughput and protocol transparency (databases, gRPC streaming, anything non-HTTP), and L7 when you need to make decisions about the content of requests. Most real architectures use both — an L4 device at the edge spreading across a fleet of L7 proxies.

Least connections is the default that survives contact with reality most often, because it self-corrects: a backend that is slow accumulates connections and is automatically given less work. Round robin does not have that property, which is why a single slow backend under round robin degrades a fixed share of all requests indefinitely.

A health check has one job — to answer “should this instance receive traffic?” — and there are two ways to get it wrong, in opposite directions.

• Too shallow. /healthz returns 200 if the process is running. The database connection pool is exhausted, every real request 500s, and the load balancer cheerfully sends 100% of traffic to instances that cannot serve any of it.
• Too deep. /healthz checks the database, the cache, and three downstream services. One of those gets slow. Every instance fails its check simultaneously. The load balancer removes the entire fleet and now you have a total outage caused by a partial degradation.
• The resolution is two endpoints with different meanings, which is exactly Kubernetes' liveness/readiness split: a liveness check that only asks ‘is this process wedged and in need of a restart?’, and a readiness check that asks ‘can this instance serve right now?’ and that is allowed to depend on critical dependencies but not on optional ones.
• And a floor. Many production balancers implement ‘panic mode’ — if more than some percentage of the pool is unhealthy, ignore health status entirely and load-balance across everything. Degraded service beats no service, and this single setting has saved a great many companies from the too-deep failure above.

When you remove an instance, in-flight requests must be allowed to finish. The sequence that works is: mark unready, wait for the balancer to notice and stop sending new work, then wait for open requests to complete, then stop the process. Skip the middle step and you get a burst of 502s on every deploy, which teams often accept as normal for years.

The subtlety that catches people in Kubernetes: the moment a pod is marked terminating, two things happen concurrently — endpoints are updated, and SIGTERM is sent. Those propagate at different speeds. Adding a preStop sleep of a few seconds before the process begins shutting down closes that race, and is the standard fix for deploy-time 502s.

A CDN gives you three separable things, and it is worth being able to separate them because teams frequently buy one and only use another. Caching of static assets close to users. Edge termination, where TCP and TLS handshakes complete near the user and only a warm, pooled connection crosses the ocean — this helps dynamic, uncacheable content substantially and is the part most teams under-use. And absorption, where volumetric attacks hit a network with terabits of capacity instead of your origin.

Cache keys are where CDN work actually lives. A cache key that includes a cookie or a user-agent will have a hit rate near zero, and the team will conclude the CDN does not help. Normalising the key — strip irrelevant query parameters, vary only on what genuinely changes the response — is usually the difference between a 40% and a 95% hit rate.