Cloud Platform Engineering · Chapter 4 of 9
Cloud networking, and where the money actually goes
Three VPC models that break each other's assumptions, and the internal traffic that costs more than your internet egress.
Cloud networking is where mental models transfer worst after identity, because the three providers made genuinely different structural choices — and where the largest avoidable costs hide, because most of them are internal traffic nobody thinks of as network spend.
Each provider gives you a private network of your own inside its cloud. AWS and Google call it a VPC, a Virtual Private Cloud; Azure calls it a VNet, a Virtual Network.
Three models, three broken assumptions
| AWS VPC | Azure VNet | GCP VPC | |
|---|---|---|---|
| Network scope | Regional | Regional | Global — one network spans every region |
| Subnet scope | Bound to one availability zone | Spans every zone in the region | Regional |
| Routing | Route table per subnet | Route tables on subnet or interface | Routes are network-wide |
| Firewall | Stateful groups at the interface, plus stateless subnet lists | Stateful groups at subnet or interface | Network-wide rules with priorities |
| Cross-project sharing | Share individual subnets | — | A host network shared with service projects |
Two of those rows break people's assumptions on arrival.
A subnet is not always zonal. Engineers from the cloud where a subnet lives in exactly one availability zone plan one subnet per zone per tier, arrive on another cloud, and cannot find the equivalent. There isn't one — zone redundancy there is a property of the resource, not of the subnet. Carrying the old pattern over produces an address plan that is three times more complicated than it needs to be.
And GCP's network is global. A single VPC spans every region, which means the multi-region transit problem that dominates network design on the other two frequently does not exist at all. Architects arriving from elsewhere build a hub-and-spoke topology to solve a problem the platform has already solved.
Peering, transit, and private access
- Peering is non-transitive on all four. A peered to B and B peered to C does not give A a path to C. Every provider sells a hub product to fix that — AWS Transit Gateway, Azure Virtual WAN, GCP Network Connectivity Center, OCI Dynamic Routing Gateway — and on three of them a hub is a router with a meter on it: an hourly charge per attachment plus a per-gigabyte processing charge. OCI charges neither, and Oracle markets that as a differentiator. GCP also waives the hourly charge on the first three VPN spokes and the first three Interconnect spokes, which is enough to make a small hub free. So check the meter before assuming one; this is the row where the four genuinely diverge.
- Overlapping address ranges block peering outright rather than half-working. This is why a single organisation-wide register of allocated ranges is worth more than any amount of clever routing later.
- Private access to managed services comes in two shapes. The expensive one puts a network interface with a private address in your subnet, and charges per endpoint per hour plus per gigabyte. The cheap one is a route table entry rather than an appliance — and AWS's gateway endpoints, for S3 and DynamoDB, are completely free.
- And on Azure, Private Link endpoints fail on DNS rather than routing almost every time. The endpoint exists, the route works, and the name still resolves to the public address because the private DNS zone was not linked. If you are debugging one, start there.
Private connectivity to on-premises — what it is before what it is called
A private circuit is a dedicated physical connection between your own network and the cloud provider, arranged through a carrier at a colocation facility where the provider has a presence. You use one instead of a VPN across the internet for three reasons: predictable latency and bandwidth rather than best-effort, higher throughput than an IPsec tunnel will sustain, and often lower egress rates than internet transfer.
Three things surprise people. It takes weeks, not minutes, because a carrier has to provision a physical cross-connect — this is the one piece of cloud infrastructure that still moves at telco speed. It is a single point of failure unless you buy two, and two circuits through the same duct is one circuit with two invoices. And — the one that matters most for a compliance answer — it is not encrypted.
| Function | AWS | Azure | GCP | OCI |
|---|---|---|---|---|
| Private circuit | Direct Connect | ExpressRoute | Cloud Interconnect | FastConnect |
| The circuit itself | a connection — one physical cross-connect, tied to one Region | a circuit — one peering location + one provider + a fixed bandwidth tier, identified by a service key | a Dedicated or Partner Interconnect | a FastConnect connection, direct or via a partner |
| Logical channels on it | Virtual interfaces: private VIF, public VIF, and a third — transit VIF for Transit Gateway | Up to two peerings: private (reaches VNets via a gateway) and Microsoft (reaches M365 and PaaS public endpoints) | VLAN attachments to a Cloud Router | virtual circuits, private or public |
| What gates your reach | physical port speed, plus which kind of virtual interface (VIF) and whether a Direct Connect Gateway is attached | the circuit SKU — the purchasable tier: Standard, Premium or Local | the attachment capacity and Cloud Router configuration | the port speed and virtual-circuit type |
So the models are not the same shape. Azure bundles bandwidth into the circuit and sells reach as a tier; AWS sells a port and makes you compose reach from interface types. Neither is better, but a design that assumes one and is built on the other ends up with an unexpected bill or an unreachable region.
None of the four encrypts a private circuit by default. Not one.
AWS states plainly that Direct Connect does not encrypt traffic in transit by default. Azure's ExpressRoute private peering is private but not encrypted at the network layer. GCP's Cloud Interconnect MACsec is off until you configure it, and even then protects only the segment up to Google's edge. OCI FastConnect is likewise unencrypted by default.
In every case the options are the same two: MACsec, where the provider and your port support it, or an IPsec overlay running across the private circuit.
Which is why “we use a dedicated connection” does not, by itself, satisfy an encryption-in-transit control — and it is exactly the kind of gap an RBI IT-outsourcing audit probes. Being able to say this unprompted is a strong signal.
And the India reality, which the announcements do not tell you
All six of AWS's India Direct Connect locations terminate in ap-south-1 (Mumbai) only — Equinix Mumbai (two), and STT facilities in Chennai, Delhi, Hyderabad and Kolkata. Note what that means: there is a physical Direct Connect location in Hyderabad, and it routes to the Mumbai region, not to the Hyderabad region sitting beside it. ap-south-2 has no native Direct Connect termination; you land in ap-south-1 and extend with a Direct Connect Gateway.
GCP is the contrasting case: both asia-south1 (Mumbai) and asia-south2 (Delhi) have local Dedicated Interconnect colocation facilities, so neither region depends on the other for private connectivity. Azure sits in between — six India peering locations across Mumbai, Chennai and Pune, with the peering location decoupled from the VNet region it ultimately reaches, the same pattern as a Direct Connect Gateway.
Two more India-specific gaps worth carrying, because they are the sort of thing a reference architecture will not warn you about: Azure's NAT Gateway does not offer the newer StandardV2 tier in West India; and Private Link support is per-service, not blanket — Microsoft Purview's Private Link lists Central India but not South or West India. “Available in India” is not a single fact, and the region a service is missing from is usually the one you picked for latency.
One category error to avoid: asking whether CloudFront, Route 53, Front Door or Cloud DNS is “available in ap-south-2” does not parse. Those are global and edge services, not region-deployed ones — Route 53's control plane lives in us-east-1 with data-plane points of presence everywhere. Regional availability is a question about regional services.
Where the network money actually goes
Three costs that are invisible until the invoice arrives
1 · Managed NAT gateway processing. Charged per gigabyte in addition to whatever the destination charge is. Traffic from a private subnet to object storage in the same region goes through it and is billed — even though it never leaves the provider. Routing it through a free gateway endpoint instead is the single most common avoidable line on an infrastructure bill, and it takes about ten minutes.
2 · Cross-availability-zone traffic, billed in each direction. A replicated cluster spread across three zones pays the cross-zone rate twice for every replica it writes. Nobody budgeted it because nobody thinks of internal replication as egress. Topology-aware routing that prefers a same-zone endpoint is usually a configuration change rather than a redesign. Traffic that stays inside one zone over private addressing is not charged on any of them — it is crossing a zone boundary that meters.
3 · Everything through a hub. A transit hub charges per attachment per hour and per gigabyte processed, so routing all inter-network traffic through one for tidiness has a real per-byte cost.
The general lesson: the headline internet egress rate is rarely the largest network line. Internal paths usually are, and they are invisible on a per-service view of the bill because they are attributed to the network rather than to the workload that caused them.
Load balancers, and a pricing detail worth knowing
Every cloud has an L4 balancer, an L7 balancer, and a global edge product, and the selection logic from the networking book applies: L4 for throughput and protocol transparency, L7 for decisions about request content.
One capacity detail is worth knowing specifically because it inverts the usual advice. On AWS's Network Load Balancer, the capacity unit allows 800 new connections per second for plain TCP and only 50 per second when the balancer terminates TLS — a sixteenfold difference. So terminating TLS at the L4 balancer is dramatically more expensive per connection than people assume, and for workloads with many short-lived TLS connections — mobile clients, APIs without keep-alive — the supposedly cheaper L4 option can cost more than the L7 one doing the same job.
The Application Load Balancer charges a reduced byte allowance for Lambda targets, which is another 2.5× penalty that appears nowhere in the headline price. The general point: load balancer pricing is dimensional, and the dimension you are actually consuming is rarely the one you assumed. Read which dimension is binding before optimising the wrong one.
What to check in a design review
- Is the address plan registered organisation-wide? Overlaps block peering permanently and the fix afterwards is renumbering.
- Is object-storage traffic on a free endpoint or through a metered gateway?
- Is anything chatty spread across zones for no reason? Redundancy needs cross-zone; a cache talking to its own application does not.
- Are private endpoints resolving privately? Check the DNS, not the route.
- And is diversity physical? Two circuits from two providers through the same duct is one circuit with two invoices — which is the same lesson as the networking book and is repeated here because people keep buying it.
Check yourself