Lakshya

Networks, Automation, Connectivity & Data Centres · Chapter 4 of 18

Routing, and how the internet decides where your packet goes

From a static route to BGP. Including why the internet occasionally routes half of Europe through a small ISP by accident.

7 min read1 diagramAll 18 chapters

Routing is the process of choosing the next hop. Inside your own network you control it; between networks it is a negotiation between around 75,000 independent organisations who mostly trust each other, which is a design decision from the 1980s that we have never managed to replace.

The route table, and the one rule that governs it

A router holds a table of prefixes and next hops. For each packet it finds every matching prefix and picks the longest one — the most specific. A /24 beats a /16 beats a /8 beats the default route 0.0.0.0/0. Nothing overrides this: not administrative distance, not metric, not protocol preference. Those only break ties between routes of the same prefix length.

Once you hold that firmly, several things become obvious. Why a single more-specific static route can black-hole traffic that everything else routes correctly. Why adding a /32 is the standard trick for pinning one host down a different path. And why route hijacking works.

Interior protocols: OSPF, in the depth an interview wants

  • Static routes — you type them. Fine for small, stable, or deliberately-pinned paths. They do not react to failure, which is either the point or the problem.
  • OSPF — a link-state protocol. Every router floods its local view, everyone builds an identical map of the area, and each runs Dijkstra against it. Converges fast, scales to a few hundred routers per area, cost defaults to a function of bandwidth. Area 0 is the backbone and all other areas must touch it.
  • IS-IS — the same class of protocol, less common in enterprises, very common in large service providers because it scales better and is not tied to IP.
  • EIGRP — Cisco's, now nominally open. Fast convergence, still widely deployed in Cisco-heavy enterprises.

The interview-useful contrast: link-state protocols share a map and each router computes its own path; distance-vector protocols share their conclusions and trust their neighbours. That difference is why link-state converges faster and why distance-vector protocols need loop prevention mechanisms like split horizon.

BGP, which is a different kind of thing entirely

BGP is not really a routing protocol in the same sense. It is a policy protocol. It does not try to find the fastest path — it has no idea what fast means, no latency measurement, no bandwidth awareness. It finds a path that everyone in the chain has agreed to carry, which is a commercial question as much as a technical one.

NORMAL: THE SHORTER AS PATH WINS AS 100you AS 200 AS 300+AS 400 AS 500owns /16 via AS200: path = 200 500  (2 hops) ← CHOSEN via AS300: path = 300 400 500 (3 hops) Path length is only tie-break #4. Localpreference, set by you, outranks it — which is how yousteer traffic. HIJACK: A MORE SPECIFIC PREFIX BEATS ANY PATH LENGTH AS 500 advertises 203.0.113.0/16 legitimate holder, correct origin AS 666 advertises 203.0.113.0/24 — and wins everywhere longest prefix match is checked BEFORE any path attribute This is not a bug. BGP has no notion of who owns an address — it is a network of mutual trust between around 75,000 autonomoussystems. RPKI/ROA is the fix: the holder cryptographically signs “AS 500 may originate this prefix”, and validating networks drop the invalidadvertisement.
Path selection, and the reason hijacks work. A shorter AS path wins a tie, but a more specific prefix does not even enter the tie-break — longest prefix match happens first. BGP has no built-in notion of who is entitled to announce an address.

The decision process, in order, with the ones that matter in practice marked:

#AttributeWho sets itNote
1Highest weightYou, locally (Cisco-specific)Never leaves the router
2Highest local preferenceYou, network-wideThe main tool for choosing your outbound path
3Locally originated—Prefer your own routes
4Shortest AS pathThe path itselfThe one everybody quotes; only the fourth criterion
5Lowest origin typeOriginatorIGP < EGP < incomplete
6Lowest MEDYour neighbourA hint about which of their entry points to use — often ignored
7eBGP over iBGP—External beats internal
8Lowest IGP cost to next hopYour IGP‘Hot potato’ — hand it off as soon as possible

The practical summary: local preference controls what leaves your network; AS path prepending is the crude tool you use to influence what enters it. If you announce the same prefix on two links and want most traffic on one, you prepend your own AS number two or three times on the other, making it look further away. It is inelegant and it is what everyone does.

Peering and transit — the commercial layer

  • Transit — you pay someone to carry your traffic to the whole internet. Billed on the 95th percentile of measured throughput, which means you can burst for 36 hours a month for free and the shape of that billing drives real engineering decisions.
  • Settlement-free peering — two networks exchange traffic directly, for free, because it saves both of them transit costs. Only covers each other's customers, never the whole internet.
  • Paid peering — when the traffic ratio is lopsided enough that one side wants payment. This is what most large content-to-eyeball disputes are actually about.
  • An internet exchange (IXP) — a shared switch fabric where hundreds of networks peer with each other over one port. You pay for the port, not per gigabit, which is why exchanges are the cheapest bandwidth on the internet.

The two failure modes worth being able to describe

Route leak. A network that buys transit from two providers accidentally re-announces one provider's routes to the other, advertising itself as a path between two much larger networks. Traffic that should never touch it suddenly does, and it has neither the capacity nor the intent. This has repeatedly taken large portions of the internet offline for tens of minutes. The defence is prefix filtering and maximum-prefix limits on every session — if a neighbour who normally announces 30 prefixes suddenly announces 200,000, drop the session automatically.

Prefix hijack. An AS announces a more specific prefix it does not hold. Because longest prefix match is evaluated before any policy attribute, it wins globally and instantly. This has been used to intercept cryptocurrency traffic and to issue fraudulent TLS certificates by hijacking the validation path. The defence is RPKI: the address holder publishes a signed Route Origin Authorisation, and validating networks drop announcements that contradict it. Adoption passed a majority of routes in the mid-2020s and is one of the genuinely good news stories in internet security.

What this looks like in a cloud interview

You will rarely run BGP yourself in a cloud role, but you will meet it constantly at the edges: Direct Connect and ExpressRoute are BGP sessions, anycast for a global load balancer is BGP, and your CDN's entire routing model is BGP.

The answer that shows understanding: “BGP chooses a policy-compliant path, not a fast one, which is why a packet from Mumbai to Singapore sometimes goes via Europe — and why you cannot fix latency by asking the network team nicely. You fix it by changing where you terminate, not by changing the path.”

Check yourself

Why is BGP called a path-vector protocol?It advertises the full list of autonomous systems a route passes through, which is both its loop-prevention mechanism and its main tie-break.
How do you influence traffic coming into your network?AS path prepending, MED, or selective announcement. All are hints — the far end makes the final decision, so inbound control is always weaker than outbound.
A /24 hijack beats your /16. Why can't BGP just reject it?BGP has no ownership model. Longest prefix match is evaluated before any attribute. RPKI adds the missing ownership assertion cryptographically.
What is 95th percentile billing and how do you exploit it legitimately?Your throughput is sampled every five minutes for a month and the top 5% of samples are discarded. You get roughly 36 hours of unbilled burst — so schedule backups and large transfers into it.
← Switching, VLANs and the layer-2 worldDNS, properly →