Networks, Automation, Connectivity & Data Centres · Chapter 12 of 18
Diagnosing a network problem, with four worked scenarios
A ladder that bisects the stack, the eight commands that settle each rung, and four real problems worked end to end.
The difference between someone who is fast at network problems and someone who is slow is almost never knowledge. It is method. Slow debuggers start at the layer they are most comfortable with — usually the application — and sweep. Fast ones bisect: pick a rung in the middle, get a definitive answer, and eliminate half the stack.
The commands, and what each one actually proves
| Command | Proves | Does not prove |
|---|---|---|
ping | ICMP reaches the host and it replies | That any TCP service works — many hosts deliberately drop ICMP |
mtr -rwzbc100 | Loss and latency per hop over 100 probes | That loss at hop 4 is real — routers deprioritise ICMP to themselves. Only loss that continues to the final hop matters |
dig +trace | The authoritative answer, bypassing every cache | What your application's resolver returns |
nc -vz host 443 | A TCP listener accepts on that port | That the service behind it is healthy |
ss -tanp | Local socket states and who owns them | Anything about the far end |
curl -v -w timings | Where the time goes: DNS, connect, TLS, first byte | Which backend served you, unless you log it |
tcpdump -ni any port 443 | Exactly what is on the wire | Anything, if you capture on the wrong interface — a very common waste of an hour |
openssl s_client | The chain the server actually presents | What a browser would accept, since browsers are more forgiving |
The single most useful line in this book
curl -o /dev/null -s -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://host/path
This decomposes a slow request into its phases in one command. Fast connect and slow first byte means the application is slow, and the network is exonerated. Slow connect means the network. Slow name lookup means DNS. Running this before you open a single log file will settle the majority of ‘the network is slow’ arguments you will ever have.
Scenario 1 — “The API is slow from the Bengaluru office only”
Claim. One office reports slowness. Other offices and home users are fine. Same application, same time of day.
Bisect. Run the curl timing line from inside the office and from a laptop on a mobile hotspot in the same building. If the hotspot is fast, the problem is inside the office network or its upstream, and you have eliminated the application, the cloud region and the destination in one test that takes ninety seconds.
Narrow. mtr from the office to the destination. Look for the hop where latency steps up and stays up for every subsequent hop. A single hop showing loss with clean hops after it is ICMP deprioritisation and is not your problem.
Common causes, in the order they turn out to be true. A saturated office uplink (check the 95th percentile utilisation graph, not the instantaneous one); a proxy or inspection appliance that is CPU-bound at peak; a change in the ISP's upstream path adding an intercontinental leg; or an MTU problem via a newly-added tunnel.
What makes this a good interview answer. The hotspot test. It is cheap, it is decisive, and it demonstrates bisection rather than sweeping.
Scenario 2 — “Uploads over 2 MB hang, small ones are fine”
Claim. The API works. Large payloads hang and eventually time out. Reproducible, and only from certain networks.
Recognise the signature. Size-dependent failure with a clean cutoff is almost always path MTU discovery being black-holed. Small packets fit; large ones need fragmentation, the router sends ICMP ‘fragmentation needed’, a firewall drops that ICMP, and the sender never learns to reduce its size.
Confirm. ping -M do -s 1472 host sends a 1500-byte packet with don't-fragment set. If that fails while a plain ping succeeds, you have proved it. Bisect the size down to find the actual working MTU.
Fix. Properly: stop dropping ICMP type 3 code 4 — it is not a security control, it is a required part of IP. Practically, if you cannot change the firewall: clamp MSS at the tunnel endpoint (--clamp-mss-to-pmtu), or lower the interface MTU.
Why it recurs. Every VPN, VXLAN overlay and PPPoE link reduces the effective MTU. Adding one to an existing path is a very common trigger, and the resulting bug report never mentions the tunnel.
Scenario 3 — “Intermittent 502s, roughly 1% of requests”
Claim. A small but persistent error rate. No pattern by user or endpoint. The application logs show nothing, because the application never received these requests.
First question: who generated the 502? A 502 comes from the proxy, not the app. So the proxy's access log is the source of truth. Look at upstream response time: if it is zero, the connection to the backend failed outright; if it is at exactly your timeout value, the backend accepted and never answered. Those are different bugs.
Correlate. Do the 502s cluster on one backend instance? Then it is that instance — check its own resource metrics. Are they evenly spread? Then it is systemic. Do they spike at deploy times? Then it is connection draining, and the fix is the preStop hook from the load-balancing chapter.
The one people miss. Keep-alive timeout mismatch. If the proxy holds connections open for 75 seconds and the backend closes them at 60, the proxy will periodically reuse a connection the backend has just closed — and the request fails with no application-side trace at all. The backend's idle timeout must always be longer than the proxy's. This single rule explains a remarkable number of unexplained sub-percent error rates, and mentioning it is a strong signal in an interview.
Also check conntrack table exhaustion on any NAT device in the path, and ephemeral port exhaustion on the proxy if it makes many short-lived outbound connections.
Scenario 4 — “It works from the office but not over the VPN”
Claim. Same user, same credentials, same target. Works on the corporate LAN, fails on VPN. The application shows a timeout, not a rejection.
Timeout versus refused is the first fork. Connection refused means a host answered and nothing is listening — a service problem. A timeout means nothing answered at all — routing or a firewall dropping silently.
Likely causes, in order. Split-tunnel routing, where the VPN client does not have a route for the destination and it goes out the local internet instead — check ip route get <dest>, which tells you exactly which interface will be used and is far more reliable than reading the route table.
Then split-horizon DNS: the internal name resolves to a private address only from the internal resolver, and the VPN client is using a public one. dig @internal-resolver versus dig @1.1.1.1 settles it in seconds.
Then asymmetric routing through a stateful firewall: the request goes out via the VPN and the reply comes back via a different path, so the firewall sees a reply to a connection it never saw open and drops it. This is the hardest of the four to see and the one where a packet capture on both ends earns its cost.
Then MTU again, because a VPN is a tunnel, and tunnels reduce MTU.
How to answer a debugging question in an interview
The structure that works: state what you would establish first and why it eliminates the most, then branch. Do not list twenty things you would check — that reads as a checklist rather than a method. One good sentence is: “Before anything else I'd establish whether it is DNS, connect, TLS or the application, because one curl command separates those four and each leads somewhere completely different.”
And when you have found it, say what you would change so it cannot recur. The distinction between fixing the instance and fixing the class is the thing senior interviewers are actually listening for, in this subject and in every other.
Check yourself