Infrastructure & Reliability · Reliability, operations & platforms · Chapter 3 of 33
Networks, without the certification
The layer everything blames and almost nobody measures properly. Six mechanisms explain most of what you will meet.
You do not need a routing certification to be effective here. You need to know what actually produces the symptoms you will be handed, and how to measure rather than infer.
Why ‘the network is slow’ is almost never bandwidth
| Mechanism | What it looks like | How you confirm it |
|---|---|---|
| Packet loss | Throughput collapses on long paths; small transfers fine | Retransmits in a packet capture — even 0.1% is destructive over distance |
| Micro-bursts | Interface averages look idle; users see stalls | Interface discards and queue depth, not utilisation |
| MTU / fragmentation | Small requests work, large ones hang | Test with increasing payload sizes; classic with tunnels and VPNs |
| Asymmetric routing | Works one way, breaks through a stateful device | Traceroute both directions — not just outbound |
| A middlebox | Intermittent, unexplainable | Firewall, proxy or TLS inspection. Very often the actual answer |
| DNS | Slow to start, fast once running | Time the resolution separately from the transfer |
Measure like the application, not like a diagnostic
ICMP is deprioritised or rate-limited on plenty of equipment, so a clean ping proves very little and a lossy ping may mean nothing. Test with a TCP connection to the actual port, continuously, from the affected location — latency and loss over time rather than a snapshot of a moment when it happened to be fine.
The sentence that ends most arguments
“I have discards on that interface and retransmits in the capture” is a fact. “It feels like the network” is not, and neither is “bandwidth looks fine”.
Getting a packet capture at both ends before concluding anything is the habit that separates people who resolve these from people who escalate them.
The four numbers that explain most network behaviour
| Quantity | Rough value | What it explains |
|---|---|---|
| Speed of light in fibre | ~200,000 km/s — about 5 ms per 1,000 km, doubled for a round trip | Physics you cannot optimise. London to Singapore is ~11,000 km, so ~110 ms round trip at absolute best. Real paths are longer |
| A TCP handshake | One round trip | Why connection reuse matters more than payload size on a distant link |
| A TLS handshake | One further round trip (or zero on resumption) | Why a fresh HTTPS connection to a far region costs three round trips before a byte of data moves |
| A same-datacentre round trip | Well under a millisecond | Why the same call is imperceptible locally and ruinous across regions — and why a chatty protocol survives testing and dies in production |
These four explain the single most common architectural performance failure: a design that makes many small sequential calls works perfectly in a datacentre where each costs a fraction of a millisecond, and collapses across a region boundary where each costs fifty. Chattiness is not a latency problem locally and is the entire problem remotely.
- Count round trips, not bytes, whenever a boundary is crossed. Bandwidth has grown enormously; latency is bounded by physics and has not.
- Latency and throughput are different problems with different fixes. A fat pipe does not make a distant call fast; it makes a large transfer fast once it starts.
- Bandwidth-delay product is why a long fat link underperforms without window tuning: you cannot fill a high-latency link with a small window, no matter what the capacity is.
- The tail is where users live. Packet loss triggers retransmission, so a link with 1% loss has a p99 far worse than its average suggests — and averages of network latency are almost always misleading.
Reading a network problem quickly
Slow but working, consistently → a distance or round-trip-count problem. Count the hops in the request path and look for a boundary crossing you forgot about — a dependency in another region is the usual culprit.
Fast then suddenly slow, repeating → congestion, or a queue filling somewhere. Look at utilisation at peak rather than average.
Intermittent failures with retries succeeding → packet loss, or a timeout shorter than the real operation. Compare the client's timeout against the actual server duration; they are frequently mismatched and nobody has checked.
Works from one place, not another → routing, DNS, a firewall rule, or an expired certificate on one path. And a caution worth stating: a fast failure is usually a rejection — a certificate mismatch, a closed port, a refused connection. A slow failure that ends in a timeout is usually a blackhole — traffic going somewhere and never coming back. Conflating those two costs hours.
And the check people skip: confirm you are actually testing from a different network. Two machines behind the same egress point are one vantage point, not two, and every conclusion drawn from them is about that path only.
Check yourself