Infrastructure & Reliability · Reliability, operations & platforms · Chapter 11 of 33
Kubernetes, actually
Not the tutorial. The control loops, the scheduler's real algorithm, and why your pod is Pending.
Kubernetes is usually taught as a set of YAML shapes, which is why so many people can deploy to it and so few can debug it. The whole system is one idea repeated: a controller watches desired state, observes actual state, and takes one step to close the gap. Everything else follows from that.
kubectl apply. Eight components, none of which is kubectl, each a reconciliation loop rather than a procedure — which is why debugging means finding the loop that stopped closing.The control plane, component by component
- API server — the only thing that talks to etcd. Handles authentication, authorisation (RBAC), and admission control. Admission webhooks run here, synchronously, which is why a slow or broken webhook makes the whole cluster appear to hang: every write waits on it. A webhook without a sane
failurePolicyand timeout is one of the most effective ways to take a cluster down. - etcd — a consistent key-value store, and the sole source of truth. It is extremely sensitive to disk latency because every write is an fsync in a Raft consensus round. Slow etcd disk means slow everything, and it presents as unrelated timeouts all over the cluster. Put etcd on the fastest disk you have and watch its fsync duration as a first-class metric.
- Controller manager — dozens of independent loops: Deployment, ReplicaSet, Node, Job, Endpoint, and more. Each one watches, compares, acts.
- Scheduler — decides which node a pod runs on, writes a binding, and then has no further involvement. It does not start anything.
- Kubelet — on each node, watches for pods bound to it, calls the container runtime through the CRI, mounts volumes through the CSI, sets up networking through the CNI, and reports status back.
- kube-proxy — programs iptables or IPVS rules so a Service's cluster IP load-balances to pod IPs. There is no proxy process in the data path; it is packet rewriting in the kernel, which is why a Service has no logs and no latency of its own.
The scheduler's actual algorithm
Two phases, and understanding them explains almost every scheduling complaint.
- Filter — eliminate nodes that cannot run the pod. Insufficient allocatable CPU or memory for the pod's requests; taints the pod does not tolerate; node selectors and affinity that do not match; volume topology constraints; ports already taken. If every node is filtered out, the pod stays Pending.
- Score — rank the survivors. Spread across zones and nodes, prefer nodes that already have the image, balance resource use, honour pod affinity and anti-affinity preferences.
- Bind — write the chosen node onto the pod.
The scheduler only ever looks at requests, never at actual usage
This is the single most important fact about Kubernetes capacity and the one that most surprises people.
A node showing 20% actual CPU use can be completely unschedulable, because the sum of the requests of the pods on it has consumed all allocatable CPU. The pods requested far more than they use, which is extremely common, and the scheduler has no idea.
The consequences: a cluster can be simultaneously full and idle; the fix is right-sizing requests rather than adding nodes; and a Vertical Pod Autoscaler in recommendation mode is usually the fastest way to find tens of percent of reclaimable capacity. Being able to explain this distinction is worth more in an infrastructure interview than knowing any amount of YAML.
QoS classes, and who dies first
| Class | How you get it | What it means under pressure |
|---|---|---|
| Guaranteed | requests == limits, for both CPU and memory, on every container | Evicted last. Use for anything stateful or latency-critical |
| Burstable | requests set, limits higher or absent | Evicted after BestEffort. The common case |
| BestEffort | no requests and no limits | Evicted first, and scheduled anywhere. Fine for batch, wrong for anything that matters |
Under node memory pressure the kubelet evicts in that order, and within a class it prefers pods using most above their requests. So a pod that requested nothing and is using a lot is the first to go — which is usually correct and is a nasty surprise if nobody set requests.
Probes: three of them, and conflating them causes outages
- Startup probe — ‘has it finished booting?’ While it is failing, the other probes are suspended. This exists precisely so slow-starting applications do not need a liveness probe with a huge threshold, which would then be useless later.
- Readiness probe — ‘should it receive traffic right now?’ Failing removes the pod from Service endpoints. It does not restart anything. This is the one that should check critical dependencies.
- Liveness probe — ‘is it wedged and in need of a restart?’ Failing kills the container. A liveness probe that checks a downstream dependency is an outage generator: the database gets slow, every pod's liveness fails at once, the whole fleet restarts, the restarts add load, and you have converted a degradation into an outage. Liveness should test only the process's own health — can it serve a trivial request — and nothing else.
Networking, in the amount you need
Every pod gets a routable IP, and pods can reach each other directly without NAT. That is the model, and the CNI plugin implements it — by overlay (VXLAN, as in Flannel or Calico's overlay mode), by native routing (Calico BGP), or by handing out real VPC (Virtual Private Cloud) addresses (the AWS VPC CNI).
The third choice has a specific and common failure: your pods consume real subnet addresses, so a large node pool can exhaust the subnet and pods stop getting IPs. ‘Pending with a CNI error’ on a cluster with plenty of CPU is nearly always this. It is a capacity-planning problem in the address plan, and it belongs in the design review rather than in the incident.
Two more worth knowing. DNS: CoreDNS with a default ndots:5 search path means an external lookup like api.example.com can generate several failed queries before the right one — a well-known and easily-fixed source of latency, either with a fully-qualified name ending in a dot or a tuned dnsConfig. And NetworkPolicy is default-allow until a policy selects a pod, at which point that pod becomes default-deny for the direction covered. Half-written policies therefore break things in ways that look random.
Debugging, in order
Pending — the scheduler could not place it. kubectl describe pod and read the events; they name the reason explicitly, such as insufficient CPU on every node or an unsatisfied node selector. Check requests against allocatable, not against usage.
ContainerCreating — the kubelet has it and is stuck. Almost always volume attachment or the CNI. Events will say.
ImagePullBackOff — registry credentials, a wrong tag, or a rate limit. Events name it.
CrashLoopBackOff — it started and exited. kubectl logs --previous is the command, and forgetting --previous is why people say ‘there are no logs’.
OOMKilled — it exceeded its own memory limit, which has nothing to do with node memory. Check lastState.terminated.
Running but not serving — readiness failing, so it is not in the Service's endpoints. kubectl get endpoints is the fastest confirmation, and an empty endpoint list with running pods is unambiguous.
The general rule: kubectl describe before kubectl logs. The events tell you which loop stopped closing, and that is the question.
Check yourself