Networks, Automation, Connectivity & Data Centres · Chapter 16 of 18
Changing a network without losing it
The outage data, the control that recovers a change which cut your own path, and a pipeline that tests before it pushes.
Network change is where automation pays and where it bites, and the data is unusually clear about which.
Two more figures from the same source: 92% of respondents say human error contributed to their most recent impactful outage, with ‘failure to follow procedures’ the leading mechanism at 59%. And the researchers' own framing is worth quoting — automation is improving flexibility and recovery, but also increasing the risk of configuration errors and cascading failures, so reliability now depends not only on reducing failures but on limiting how far and how quickly they propagate.
The controls, in the order they earn their place
- Commit-confirmed with an auto-revert timer. The device rolls back automatically unless you re-confirm within a window, so a change that severs your management path heals itself. This is the single highest-value control in network operations and the one with no server equivalent. The gap to name: not every platform supports it, and the ones that do not are usually the old campus switches carrying your out-of-band path.
- Batch by failure domain, never by percentage. One device in the least-critical domain, then one per domain, then one full domain, then the fleet. ‘Five percent of devices’ is meaningless if that five percent is both spines.
- Never both halves of a redundant pair in one change window — encoded as a hard gate in the pipeline, not a line in a runbook.
- Automated health gates between stages. Post-change state assertions must pass before the next batch proceeds, and a failure halts and rolls back.
- Out-of-band access as a first-class dependency — separate path, separate power, separate credentials with a local fallback for when the authentication server is unreachable. And tested on a schedule: an untested out-of-band path is a belief, not a control. ‘When did you last verify you could reach a core switch console with the production network down?’ separates people who have run networks from people who have read about them.
A pipeline that tests before it pushes
| Layer | What it does | Cost |
|---|---|---|
| Schema validation | Is the source-of-truth data even valid? | Seconds |
| Offline render | Produce the config text with no device involved; diff against golden files | Seconds |
| Static analysis | Parse configs into a vendor-independent model and compute the control and data plane offline — then assert reachability and policy, and diff the result against the current config | Minutes |
| Emulation | Boot the real network OS images in containers, apply the candidate, assert adjacencies and forwarding | Minutes |
| Human review | Of the rendered config diff, per device class — not the YAML | Minutes |
| Device dry-run | Commit-check or compare-config on the device itself | Seconds — and it validates syntax, not impact |
Two things almost every real pipeline gets wrong.
Reviewing the wrong artefact. Humans review the template and variable diff. They must review the rendered config diff, because a one-line variable change can produce a nine-hundred-line config delta and nobody can hold that mapping in their head.
Asserting the wrong property. ‘The routing protocol is up in the candidate’ is weak. The property you actually care about is differential: the set of reachable flows in the candidate is identical to the current set except for these twelve intended additions. Static analysis tools give you exactly that and it is the single most valuable assertion available.
A caution on the static analyser: its coverage is bounded by its parsers. Unparsed lines are silently absent from the model, so an assertion can pass because the relevant configuration was never understood. Always read the parse warnings — a clean run with four hundred unparsed lines is a false-negative machine.
And on emulation: containerised network operating systems are control-plane accurate and data-plane approximate. They will catch a broken policy, a missing adjacency or a config that fails to commit. They will not catch hardware table exhaustion, vendor-specific forwarding behaviour, optics problems or scale-dependent convergence.
Industrial precedent, for anyone who thinks this is theoretical: one hyperscaler has run automated access-list verification in production since 2013, and published the design — local contracts checked locally and in parallel, without maintaining a global snapshot, used to establish global properties. Another studied more than a hundred high-impact failures across its data centres and wide-area networks and reported that a large number occurred while a network management operation was in progress. The risk is not the change you are afraid of; it is the routine one.
Check yourself