Infrastructure & Reliability · Chapter 8 of 8
Kubernetes, honestly
What it genuinely solves, what it costs, and the four things that break in production.
Kubernetes appears in 38% of platform postings and 26% of IT infrastructure ones, so it is worth understanding properly — including being able to say when it is the wrong choice, which is a stronger interview signal than enthusiasm.
What it actually gives you
A declarative API over a fleet, with a control loop that continuously reconciles actual state toward desired state. That reconciliation loop is the whole idea; scheduling, self-healing and rolling updates all fall out of it. Everything else — the YAML, the ecosystem, the operators — is machinery around that.
What it costs
- An operational surface of its own. You now run a distributed system in order to run your distributed system, and it needs upgrades, capacity and expertise.
- Debugging gains a layer. Every question now has a container, a pod, a node and a network policy between you and the answer.
- It is not a security boundary by itself. Namespaces are organisational; the kernel is shared. Untrusted workloads need node separation or a sandboxed runtime.
- For a small number of services, it is usually not worth it. Saying so is a credibility signal, not a weakness.
The four things that actually break
| Symptom | Usual cause | Where to look |
|---|---|---|
| Pod pending forever | No node satisfies the request — resources, taints, affinity, or a volume in the wrong zone | describe pod events, not the logs |
| OOMKilled | Limit set below real usage, or a limit set at all where the workload spikes | Memory limit versus actual working set; consider requests without limits |
| CrashLoopBackOff | The app fails at startup — config, secret, or a dependency not ready | Previous container's logs, and the readiness probe |
| Intermittent 5xx during deploy | Readiness probe passes before the app is actually ready, so traffic arrives too early | Probe definition and terminationGracePeriodSeconds |
Notice that three of the four are configuration rather than Kubernetes faults. That is representative: most production Kubernetes problems are requests, limits and probes set by someone who was copying an example.
Requests and limits, the one thing to get right
Requests determine scheduling — what the scheduler reserves. Limits determine enforcement — where the kernel throttles or kills. Setting requests too low overcommits the node and produces mysterious contention; setting memory limits too tight produces OOM kills under normal variance.
A defensible default: set memory requests and limits equal, so the workload is predictable and cannot be evicted for exceeding what was reserved. Be much more cautious with CPU limits, because CPU throttling produces latency that looks like an application bug and is very hard to trace.