Networks, Automation, Connectivity & Data Centres · Chapter 5 of 18
DNS, properly
A distributed cache with no invalidation, holding up the entire internet. Two of the largest outages in history were DNS.
DNS is the layer that everyone thinks they understand and almost nobody has read the details of. It is worth an hour, because a disproportionate share of production incidents are DNS, including some famous ones, and because “it's always DNS” is a joke that engineers repeat without being able to explain why it is always DNS.
The answer to that is structural. DNS is a globally distributed cache with time-based invalidation and no way to purge. Every change you make propagates on a schedule you set in the past, to caches you do not control, operated by people you will never meet. It is the only system in your stack with those properties.
The two halves people conflate
- The authoritative server holds the actual zone data for a domain. It is the source of truth. Route 53, Cloudflare, NS1, Akamai, or a BIND server you run.
- The recursive resolver holds nothing of its own. It answers on behalf of clients by walking the tree and then caching what it learns. 8.8.8.8, 1.1.1.1, your ISP's, your corporate one,
systemd-resolvedon your laptop. - Almost every confusing DNS symptom comes from these two being out of step: the authoritative answer has changed and some resolver in the chain is still serving what it cached.
The record types that matter, and their sharp edges
| Record | What it does | The sharp edge |
|---|---|---|
| A / AAAA | Name → IPv4 / IPv6 address | Multiple A records give crude round-robin with no health awareness whatsoever |
| CNAME | Name → another name | Cannot coexist with any other record at the same name, which is why you cannot CNAME a zone apex — no CNAME alongside the mandatory SOA and NS |
| ALIAS / ANAME | Apex-safe CNAME | Not a standard — a provider-side trick where the authoritative server resolves the target and returns A records. Behaviour differs between providers |
| MX | Mail exchanger, with priority | Lower number is higher priority. Removing an MX record does not stop mail instantly — senders retry for days |
| TXT | Arbitrary text | Where SPF, DKIM and DMARC live, plus every domain-ownership verification you have ever done |
| NS | Delegates a zone to nameservers | A mismatch between the parent's NS records and the zone's own is the classic half-working delegation |
| SOA | Zone metadata and the negative-cache TTL | The minimum field controls how long NXDOMAIN is cached — so a typo can be cached as ‘does not exist’ long after you fix it |
| SRV | Service location with port and priority | Used heavily by Active Directory, SIP and Kubernetes |
| CAA | Which CAs may issue for this domain | Cheap and under-used. Stops a mis-issuance by a CA you never intended to use |
The negative-cache trap, which catches everyone once
You add a new subdomain. Someone tests it thirty seconds before you finish, gets NXDOMAIN, and that negative answer is cached — for as long as the SOA minimum says, which is commonly an hour or more.
Now the record exists, is correct everywhere, and that one person still cannot resolve it. They will tell you your change did not work. It did.
This is why you create the record before you tell anyone, and why the SOA minimum should be short (300s) on zones that change.
What actually happens on a cold lookup
Your resolver asks a root server for api.example.com. The root does not know, but it knows the .com nameservers, and says so. The resolver asks one of those, which does not know either but knows example.com's nameservers. The resolver asks one of those, which is authoritative and gives the answer. Three round trips, then cached.
dig +trace api.example.com performs exactly this walk and prints every step. It is the tool that settles arguments, because it bypasses your resolver's cache entirely — if +trace gives the new answer and your normal dig gives the old one, the change is correct and you are looking at a cache. That single distinction resolves a large fraction of DNS incidents in under a minute.
Anycast, and why 8.8.8.8 answers in 4ms
A DNS root server address is announced from hundreds of physical locations simultaneously via BGP. Your packet goes to whichever one is closest in BGP terms. This is anycast, it is why there are 13 root server addresses but well over a thousand physical instances, and it is the same mechanism CDNs use for their edges.
The operational consequence worth knowing: with anycast, two people in different cities genuinely can get different answers from the same IP address, and that is not a bug. When someone reports a problem you cannot reproduce, geography is a real hypothesis.
DNSSEC, and the honest assessment of it
DNSSEC signs DNS answers so a resolver can verify they were not tampered with in transit. It does not encrypt anything — that is DoH and DoT, which are different things solving a different problem (privacy, not authenticity).
The honest position, which is a good thing to be able to give in an interview because it shows judgement rather than recital: DNSSEC solves a real problem, adoption is partial, and the operational failure mode is severe. A key rollover that goes wrong does not degrade your domain — it makes it disappear for every validating resolver, which is now a large share of them. Several national-scale outages have been exactly this. If you turn it on, you automate the rollover and you monitor the signature expiry as a first-class alert.
Two outages worth knowing by name
Dyn, October 2016. A DDoS against a managed DNS provider took out resolution for a long list of major sites simultaneously — not because those sites were attacked, but because they all delegated DNS to the same provider. The lesson that came out of it, and that is still under-applied: your DNS provider is a single point of failure for your entire company, and secondary DNS with a second independent provider is one of the cheapest resilience purchases available.
Facebook, October 2021. A configuration change withdrew the BGP announcements for the prefixes containing their authoritative DNS servers. The DNS servers were healthy; they were simply unreachable, so every name failed to resolve globally. And because their own internal tooling, badge access and remote management all resolved through the same names, engineers physically could not get into the buildings to fix it. It is the canonical example of a circular dependency in a control plane, and it is worth being able to describe because ‘does your recovery path depend on the thing that is broken?’ is a question you should be asking in design reviews for the rest of your career.
A scenario: half the users, one region
Symptom. After a migration, roughly 30% of users get errors. The rest are fine. It does not correlate with browser or device. Retrying sometimes works.
First move. dig @8.8.8.8 app.example.com, dig @1.1.1.1 app.example.com, and dig +trace app.example.com. Compare. If the answers differ, you have your answer without touching a server.
What it usually is. One of: the old TTL had not expired when you cut over; a stale record on one of several authoritative nameservers because the zone transfer failed to one secondary; or a split-horizon setup where the internal view was updated and the external view was not.
Why 30%. That is roughly the share of users whose resolver happened to have the old answer still cached, or that happened to be served by the one out-of-date nameserver. Percentages that are neither 0 nor 100 are almost always a caching or a load-balancing question, and that instinct alone will save you hours.
Check yourself