Networks, Automation, Connectivity & Data Centres · Chapter 6 of 18
TLS, certificates, and the outage that happens every year
The handshake, the chain of trust, and the four things that actually break — one of which is still the single most common self-inflicted outage in the industry.
An expired certificate has taken down payment networks, airlines, mobile carriers and at least one national health service. It is entirely preventable, it is preventable with software that is free, and it still happens somewhere significant every few months. That gap between how solvable it is and how often it happens is worth understanding, because the reason is organisational rather than technical.
What the handshake achieves, in one paragraph
Three things, and it is worth being able to separate them. Authentication — you verify the server is who it claims, via a certificate signed by someone your machine already trusts. Key agreement — both sides derive a shared symmetric key without ever sending it, using Diffie-Hellman. Confidentiality and integrity — everything after that is encrypted and authenticated with that symmetric key, because asymmetric crypto is far too slow for bulk data.
TLS 1.3, standardised in 2018, removed everything that had been broken over twenty years: no RSA key transport, no static Diffie-Hellman, no CBC-mode ciphers, no compression, no renegotiation. It also cut the handshake to one round trip. If you are asked what you would configure, the correct modern answer is short: TLS 1.2 as the floor with a restricted cipher list, TLS 1.3 preferred, and nothing below 1.2 at all.
The chain of trust, and why curl fails when the browser works
Your certificate is signed by an intermediate CA, which is signed by a root CA, which is in your operating system's trust store. The server must send the leaf and the intermediates — the root is already local and does not need sending.
If you forget the intermediates, browsers usually still work, because they cache intermediates from previous sites and can fetch missing ones via the Authority Information Access extension. curl, Java, Python's requests and most language runtimes do not. So you get the exact symptom that wastes an afternoon: the site loads perfectly in Chrome and every API client fails with an unknown-CA error.
The command that settles it in ten seconds is worth memorising:
openssl s_client -connect host:443 -servername host -showcerts— shows you exactly what the server sends, in order. Count the certificates. If there is only one, you have found your bug.echo | openssl s_client -connect host:443 2>/dev/null | openssl x509 -noout -dates -subject -ext subjectAltName— expiry, subject and SANs in one line.curl -vI https://host— the verbose output names the exact verification failure, which is more useful than the browser's generic warning.
The four failure modes, in order of how often they happen
| Failure | What you see | Root cause | Prevention |
|---|---|---|---|
| 1 · Expired leaf | Everything breaks at once, at a precise moment | Renewal was manual, or automated renewal succeeded and nobody reloaded the service | ACME automation plus a reload hook, plus an alert at 21 days that pages a human |
| 2 · Missing intermediate | Browsers fine, API clients fail | Deployed the leaf only | Test with openssl s_client, not a browser |
| 3 · Hostname mismatch | Name does not match certificate | Hostname not in the SAN list — CN alone has been ignored since 2017 | Check SANs, not CN. Wildcards cover one label only: *.example.com does not cover a.b.example.com |
| 4 · Clock skew | ‘Not yet valid’ on a valid cert | Client clock wrong — common on embedded devices and freshly-booted VMs | NTP everywhere, and treat time as a dependency |
Why expiry still happens in organisations that know better
It is almost never that nobody knew certificates expire. It is that the certificate was issued by a person who has since changed team, on a host nobody owns, for a service that was not in the inventory.
So the durable fix is not ‘remember to renew’. It is an inventory — scan your own estate from the outside, on a schedule, and record every certificate you find with its expiry and its owner. Certificate Transparency logs let you do this for free for your own domains: every publicly-trusted certificate issued for your domain is in a public log, which means you can find the ones you did not know about.
Being able to say that in an interview — that you fixed the class rather than the instance — is worth considerably more than knowing the handshake.
Certificate lifetimes are collapsing, and this changes your architecture
Maximum publicly-trusted certificate lifetime has fallen repeatedly: five years, then three, then 825 days, then 398 days, and the industry has agreed a path toward roughly 47 days by 2029. The direction is settled and the reason is sound — short lifetimes limit the damage from a compromised key and make revocation less important, because revocation has never worked reliably.
The consequence is not administrative, it is architectural. At 47 days, manual renewal is simply not viable, and any process involving a ticket, a purchase or a human copying a file will fail. Certificate management stops being a task and becomes a system: ACME, an internal CA for internal traffic, automated distribution, and a reload mechanism that does not require a restart. If you are asked what you would prioritise in a year, this is a genuinely defensible answer and one most candidates will not give.
mTLS, and where it earns its cost
Mutual TLS means the client also presents a certificate, so both sides authenticate. It is the backbone of zero-trust service-to-service authentication and it is what a service mesh gives you largely for free.
The honest trade-off: mTLS gives you strong, cryptographic workload identity that does not depend on network position — which is exactly the property you want when ‘inside the firewall’ has stopped meaning anything. It costs you a certificate lifecycle for every workload, and if you run that lifecycle badly you have simply moved your outage from the edge to the interior, where it is harder to see. Do it with short-lived automatically-rotated certificates (SPIFFE/SPIRE, or the mesh's own CA), or do not do it.
Check yourself