Cloud Platform Engineering · Chapter 2 of 9
Managed Kubernetes, compared honestly
What the control plane actually costs, why your nodes run out of addresses before they run out of CPU, and who forces an upgrade on you.
All three major clouds sell managed Kubernetes and the differences are larger than the marketing suggests — particularly in what happens when you fall behind on versions and in how quickly you run out of IP addresses.
What the control plane costs
| Control plane | The detail that matters | |
|---|---|---|
| AWS | $0.10 per cluster-hour standard | $0.60 per cluster-hour for extended support — six times the price, and it is a deliberate financial forcing function to make you upgrade. At fifty clusters that is roughly $263,000 a year purely for being late |
| Azure | Free tier costs nothing | The only one of the three with a genuinely free production-shaped control plane. What you give up is the financially-backed API-server SLA — and nothing else |
| $0.10 per cluster-hour | Flat regardless of mode or topology, with a free-tier credit that applies only to zonal and autopilot clusters |
The asymmetry in that table is under-discussed. One provider gives away the control plane and charges only for an availability guarantee; another charges six times list to let you stay on an old version. Both are pricing decisions expressing a product opinion, and knowing them is worth real money at scale.
Node management: the field converged on one idea
All three have moved toward the same model: instead of pre-declaring node groups and letting an autoscaler pick from them, a controller looks at the pending pods and provisions individual nodes shaped to fit them.
- Two of the three clouds now run the same open-source project for this, one as a managed mode and one as a managed add-on. The vocabulary is identical, which is unusually convenient for anyone moving between them.
- The third expresses it declaratively as an ordered list of preferences — try this machine family, then that one, then spot, then on-demand — with a policy for what to do when every option is exhausted. That is the cleanest expression of a fallback capacity strategy of the three, and it is worth knowing about even if you are not on that cloud.
- The advantage over group-based autoscaling is real: faster scale-up because it bypasses the scaling-group machinery, cheaper nodes because it picks the best fit rather than the nearest declared template, and active consolidation that repacks workloads onto fewer nodes.
- The failure mode to know: pod disruption budgets can block consolidation entirely. A workload with a restrictive budget prevents the controller from moving it, so expensive nodes persist and you pay for capacity the system wanted to reclaim.
The fully-managed node modes have a cost detail that is easy to miss and expensive to discover.
One provider's fully-managed mode charges roughly a 12% premium on the on-demand instance price — and it charges that premium the same whether the underlying capacity is on-demand, spot, or covered by a commitment.
So at a 70% spot discount, the management fee is around 40% of what you actually pay for the instance. The headline ‘12%’ is against a list price you are not paying.
The same mode also imposes a 21-day maximum node lifetime, reducible but not extendable, and nodes are appliances — no interactive access at all. Both are reasonable and both surprise teams migrating from self-managed nodes.
The address exhaustion that catches everyone
There are three networking models and the choice determines your address plan permanently.
The flat model gives every pod a real, routable address from the same network as your other resources. Pods are directly reachable from peered networks and from on-premises, which is genuinely useful. The cost is that maximum pods per node becomes a property of the instance type rather than of Kubernetes: a mid-size instance with three network interfaces at ten addresses each tops out around 29 pods, with most of its CPU and memory unused.
The mitigation is prefix delegation — hand each interface slot a block of sixteen addresses instead of one, multiplying density roughly sixteenfold. It is also faster, because it makes far fewer control-plane calls when starting pods. Two cautions: it is close to a one-way door, since you cannot downgrade the networking plugin afterwards without replacing every node; and mixing prefix-enabled and non-prefix nodes produces inconsistent advertised capacity, so the clean path is new node groups rather than rolling existing ones.
The overlay model puts pods on a separate address range and translates on the way out. Address space stops being a constraint entirely — hundreds of pods per node become routine — and the trade is that pods are not directly reachable from peered networks. Communication is pod-initiated only. For most workloads that is fine; for anything where an on-premises system must call into a pod, it is disqualifying.
Two address-planning facts worth carrying into a design review
The arithmetic. A 500-node cluster at 110 pods per node needs roughly 55,000 addresses on a flat model. That is a very large block for one cluster's pods, and it is why flat-model clusters routinely need secondary address ranges bolted on later.
And a specific trap on one cloud: the platform reserves several address ranges for its own use, and because two of them fall inside 172.16.0.0/12, that entire block is rejected outright as a pod range. Teams migrating an on-premises addressing plan that uses it hit this on day one, and the error message does not explain why.
The general lesson: subnet and pod-range sizing is a design-time decision that is painful to reverse. Spend an hour on it before the first cluster, not a quarter on it afterwards.
Upgrades: who forces you, and how
This is where the three genuinely differ in philosophy, and it is worth knowing before you choose, because it determines how much of your year is spent on upgrades.
| Support window | What happens at the end | |
|---|---|---|
| AWS | 14 months standard + 12 months extended = 26 months | The control plane is auto-upgraded with no advance notice. Only the control plane — your nodes stay behind. You can roll back an in-place upgrade within seven days, but not one that was forced |
| Azure | 12 months community support, then a further tier at N-3 | You keep running, in an unsupported grey zone — platform issues only, no Kubernetes component support, no security patches, no SLA, and you cannot create new clusters on that version. The provider reserves the right to upgrade you eventually |
| ~14 months, extendable by roughly ten more on a paid channel | Forced at end of support regardless of maintenance exclusions, and the control plane must take a patch at least every 90 days — enforced over your configured policy |
The characters are clear. One is transactional: pay six times list to defer, then get upgraded without warning. One is opinionated: upgrades happen, exclusions are bounded, and you are on a channel. One is permissive in a way that creates false comfort — you keep running past end of support, which feels fine, and you have quietly stopped receiving security patches for Kubernetes components.
That third case is the one to name in an interview, because it is the failure that does not announce itself. A cluster that still works is not a cluster that is still supported, and the gap between those two is where a vulnerability lives.
What each does that the others do not
- One provider only: genuinely serverless pods with no nodes at all; on-premises machines joining a cloud-hosted control plane; buying explicit control-plane scaling tiers; and version rollback within a week.
- Another only: a free control plane; long-term support as a purchasable tier; the richest networking matrix of the three; and unusually, an SLA on pod readiness rather than just on the API server — a novel guarantee shape.
- The third only: per-pod billing rather than per-node; declarative ordered capacity fallback with active migration to better node types as capacity frees; release channels as a first-class subscription; and a global network that makes multi-region clusters share one address space.
Check yourself