Networks, Automation, Connectivity & Data Centres · Chapter 17 of 18
Network observability
Why a five-minute poll conceals the failures that matter, what sampled flow can and cannot prove, and the traffic no switch will ever see.
Network monitoring and network observability differ in a way that is easy to state and hard to act on: monitoring tells you a thing you already thought to watch has crossed a line you already chose. Observability is having enough data to answer a question you did not anticipate. In networking the gap between them is mostly a sampling problem.
Streaming telemetry versus polling, as a sampling argument
The case for streaming telemetry is usually made as modernity. It is actually sampling theory, and that version is much harder to argue with.
At a five-minute poll interval you cannot observe any event shorter than ten minutes. A routing session that flaps and re-establishes in forty seconds is invisible — both polls return ‘up’. A two-hundred-millisecond microburst that fills a buffer and drops four thousand packets is flattened into a barely perceptible bump in a five-minute average. Averaging does not merely miss short events; it actively conceals them — and short events are exactly what breaks applications.
The fix is not a faster poll. It is a subscription that delivers the transition itself rather than sampling around it, which is what an on-change subscription in a streaming-telemetry protocol gives you. Push instead of pull, a typed path against a schema instead of an integer against a numeric identifier you have to map.
The honest adoption picture, though: polling remains near-universal and is supported by every device you own, including the ones from 2011. Streaming adoption has been slowed by fragmentation — different vendors, and within one vendor only some platforms and versions. The practical answer for nearly every real network is both: streaming for high-rate counters and session state on modern platforms, polling for the long tail of old campus gear, power and environmental sensors and third-party appliances. Run a coverage programme, not a migration.
Flow data, and the reasoning error to stop making
Sampling is unbiased for volume and catastrophic for existence. That one sentence prevents most misuse of flow data.
A one-in-a-thousand sample estimates a ten-gigabit elephant flow accurately. It has roughly a 63% chance of never seeing a thousand-packet transfer at all.
So: use sampled flow for capacity, traffic engineering, peering and volumetric attack detection. Never use it to prove a specific connection did not occur. ‘There is no flow record, therefore it did not happen’ is invalid under sampling, and it is the single most common flow-data reasoning error — usually made during a security investigation, which is the worst possible time.
Two more artefacts worth knowing. A long flow is exported in chunks on an active timeout, so naively summing records inflates flow counts and distorts duration statistics. And packet-sampling protocols export periodic interface counters alongside the samples, which is why their totals can be right while the per-flow detail is incomplete.
And the boundary of the whole data type: flow tells you that, never what. A permitted record means the packet passed a layer-3 or layer-4 check. It says nothing about whether the handshake completed, whether the session negotiated, or whether the application returned an error.
eBPF, and the traffic no switch can see
Kernel-level instrumentation changed what is observable inside a host, and one consequence dominates the rest:
- Identity instead of addresses. In an orchestrated environment addresses are ephemeral and reused within seconds, so a flow record naming two of them is nearly useless. Attaching workload, service and namespace identity at the kernel is not ‘better flow data’ — it is the only thing that makes flow data mean anything in a cluster.
- The policy verdict travels with the flow. Forwarded or dropped, and the reason — dropped by policy, dropped by connection tracking, unroutable. Traditional tooling gives you a missing packet and a guess.
- Pre-translation truth. You see the connection as the workload made it, not as it looked after address translation and encapsulation rewrote it.
- Layer 7 without a sidecar or a code change — name resolution, request method and status, and database statements, captured at kernel hooks.
- And the one to remember: east-west traffic between two workloads on the same host never touches a switch. No tap, no mirror, no flow exporter can ever see it. That is a permanent blind spot in the traditional model, and kernel instrumentation is the only thing that closes it.
The cost is real: sensitivity to kernel versions, a debugging story that requires kernel literacy, and a cardinality problem — identity-labelled flows generate an enormous label space and will destroy a metrics budget if you route them somewhere priced per series. Worth it, and not free.
Why passive monitoring is structurally insufficient
You do not own the path. Between your user and a software-as-a-service provider sit a dozen networks you have no telemetry from and never will. Passive monitoring at your edge tells you your edge is healthy; it cannot distinguish ‘the internet is broken in one city’ from ‘the provider is degraded’ from ‘our egress is fine’. Only active probing from many vantage points, with hop-by-hop measurement, separates those.
And no traffic means no data. Passive tools are silent about a service until a user hits it, so the first report of a broken path is a support ticket. The failure mode to know: a synthetic measures the synthetic transaction, not your users. A green probe and angry users usually means the probe's path, resolver, authentication flow or client profile differs from theirs. Always ask what the probe actually did.
What you genuinely cannot see in a cloud network
| Blind spot | Why it matters |
|---|---|
| No payload, ever | There is no cloud equivalent of 'just tap it' |
| Permitted ≠ succeeded | A flow log's accept means it passed a security-group check. The connection may have reset, timed out, or returned an error |
| Managed-service internals | Traffic mirroring generally cannot be applied to load balancers or serverless functions — which is exactly the tier fronting your applications |
| The provider backbone | Inter-zone and inter-region paths expose no per-hop telemetry. You get a service level agreement, not a traceroute |
| Aggregation windows | Flow logs are aggregated over an interval and can drop records under load — the same concealment failure as five-minute polling |
| Name resolution | Resolver logs are a separate product. A large share of 'network' incidents in cloud are name-resolution incidents, invisible in the dataset people reach for first |
The correlation problem, and why it is really the source-of-truth problem
An application team says checkout is slow. You must decide whether a network event caused it. That requires joining datasets with different identifiers (traces and service names on one side; addresses, interfaces and autonomous-system numbers on the other), different time granularity (microseconds against five-minute averages), different owners, and no causal arrow.
What actually works today is assisted manual pivoting, not correlation: a shared set of join keys so an engineer can move from an application symptom to a network view without rebuilding the timeline by hand.
The highest-yield automated correlation available is not machine learning — it is change-event overlay. Put configuration-change events on the same timeline as performance. Given that configuration and change management causes about 41% of network outages, “what changed in the last thirty minutes” has a better prior than any model on the market.
And here is why it stays unsolved: the join is not a data-engineering problem, it is an ontology problem. There is no shared model of which network element supports which application dependency — and the only place that model could live is a source of truth mapping services to infrastructure, which most organisations do not have.
The correlation problem is the source-of-truth problem wearing an observability hat. Sources of truth, safe change and correlation are one problem, not three.
Check yourself