Lakshya

SDLC, DevOps & Developer Experience · Chapter 2 of 15

Infrastructure as code, and the state file that runs everything

Declarative versus imperative, why the plan is the artefact you review, and the three failure modes that are all really about state.

8 min read1 diagramAll 15 chapters

Infrastructure as code is not primarily about automation — scripts automated things perfectly well. It is about making infrastructure reviewable, reproducible and auditable, which are properties of version control rather than of automation. Once you hold that, the design decisions make sense.

Declarative and imperative are different tools, not rivals

DeclarativeImperative
You writethe desired end statethe steps to get there
The tool computesthe difference and how to close itnothing — it runs what you wrote
Re-running itconverges to the same statemay or may not be safe
Typical useprovisioning cloud resourcesconfiguring an existing machine, orchestrating a sequence
The catchthe model must cover what you need, or you reach for an escape hatchidempotency is your responsibility, and it is easy to get wrong

The practical division most estates land on: declarative tooling provisions the resources, and a configuration or orchestration tool handles what happens inside and across them. The two are complementary and the argument about which is ‘better’ usually means the arguers have different problems.

One thing worth understanding about the imperative side: idempotency is a property you have to design for. A configuration tool's modules are written so that ‘ensure this package is installed’ is safe to run a hundred times, and the moment you drop to running a raw shell command you have left that guarantee behind. A playbook full of shell commands is a script with extra syntax.

The loop, and the state file

THE DECLARATIVE LOOP — AND WHY THE STATE FILE IS THE WHOLE DESIGN 1 · config what you WANT declarative, in git 2 · state what it THINKS exists the resource-to-id map 3 · refresh what ACTUALLY exists reads the live API 4 · plan the diff of all three the review artefact 5 · apply make it so the only mutating step The plan is the thing to review, not the code. Code says what you intended; the plan says what will happen to production — and theydiffer more often than anyone expects. THREE WAYS IT GOES WRONG, ALL OF THEM ABOUT STATE drift someone changed it in the console the plan now wants to undo their fix detect drift on a schedule and make console access read-only state corruption / loss the file is the ONLY link to real resources lose it and the tool forgets they exist remote state, versioned, locked import can rebuild it — slowly secrets in state the state file records resource attributes including generated passwords, in clear encrypt at rest, restrict access treat state as a secret store The blast radius question that decides your layout: one state file for everything means one lock, one plan, and one mistake that candestroy production. Split by blast radius and by change cadence — network separate from compute separate from application — and reference across theboundary by data lookup rather than by sharing one state. The counterweight: too many small states and a single change needs six coordinated applies in the right order. Both failure modes arereal; the split is a judgement, not a rule.
The declarative loop, and the three failure modes — all of which are about state. Note the review point: the plan, not the code, is what tells you what will happen to production.

The state file is the design decision that everything else follows from. It maps the resources in your configuration to the real objects that exist, by their provider-assigned identifiers. Without it the tool has no way to know that web_server in your code is that particular instance in the account — and it is not derivable, because names are not unique and tags can be edited.

1 · Drift. Someone changes something in the console during an incident, reasonably, to fix a problem. The state file no longer matches reality, and the next plan proposes to undo their fix. Detect drift on a schedule rather than discovering it at apply time, and make console access read-only for anything managed as code — with a documented break-glass path, because forbidding emergency changes just means they happen anyway and nobody tells you.

2 · State loss or corruption. The state file is the only link between your code and the real resources. Lose it and the tool believes nothing exists and will happily create a second copy of your production estate. Remote state, versioned, encrypted, with locking so two people cannot apply at once. Recovery through import is possible and it is slow and manual, so the backup matters more than the recovery procedure.

3 · Secrets in state. The state file records resource attributes, and that includes generated passwords, keys and connection strings, in clear text. This surprises people every time. Treat the state file as a secret store: encrypt at rest, restrict read access as tightly as you would a vault, and never commit it. The better pattern is to have the infrastructure code create a secret reference and let the secret itself be generated and held elsewhere.

The plan is the review artefact

This is the practice that separates teams who trust their infrastructure automation from teams who are frightened of it. Review the plan, not only the code.

Code expresses intent. The plan expresses what will actually happen to production, and the two differ more often than anyone expects — because of drift, because a module version moved, because a data source resolved differently, or because a provider changed a default. A pipeline that posts the plan onto the pull request, so a human reads the diff before approving, catches a whole class of accidents.

And the single most important thing to look for in a plan is destroy and replace. Many resource changes are updates in place; some force replacement, and for a database or a stateful volume that means deleting the real thing. The word to search for is ‘forces replacement’, and a plan containing it against anything stateful should stop the pipeline and require a second reviewer.

How to lay out state, which is a blast-radius decision

One state file for the entire estate means one lock, one plan, one apply — and one mistake that can take down everything. Splitting reduces blast radius and introduces coordination cost.

  • Split by blast radius and by change cadence. Networking changes rarely and is catastrophic to break; application infrastructure changes daily and is cheap to fix. They do not belong in the same apply.
  • Reference across boundaries by data lookup, not by sharing state. Have the downstream configuration query for the network it needs rather than reaching into another state file, so the coupling is a read rather than a shared mutable object.
  • The counterweight is real: too many small states and a single logical change requires six coordinated applies in the correct order, which is its own kind of fragility and which people underestimate. The split is a judgement about your organisation, not a rule.
  • Environments should be the same code with different inputs, so that ‘it works in staging’ means something. A separate copy of the code per environment guarantees they will diverge, and the divergence will be discovered during a production incident.

Modules, and the abstraction trap

Modules are how infrastructure code is reused, and the failure mode is over-abstraction. A module with forty input variables, half of which exist to handle one team's exception, is harder to understand than the resources it wraps and impossible to change safely.

The test worth applying: can someone read the module and predict what it creates? If it takes longer to understand the module than to write the resources directly, the abstraction is costing more than it saves. Small, composable modules that do one thing beat one module that does everything conditionally — and pinning module and provider versions explicitly is what stops a rebuild eighteen months later producing a different estate.

Policy as code, which is where this becomes a control

The last step is enforcing rules on the plan before it applies: no public storage buckets, encryption required, mandatory tags, no unrestricted security-group rules, cost thresholds requiring approval.

This is the mechanism that converts infrastructure standards from documentation into something that cannot be violated — and it is a much better answer to ‘how do you enforce cloud standards’ than a review checklist, because the check runs on every change without a human being available. The related pattern is drift detection running on a schedule and raising a ticket rather than silently correcting, so a human decides whether the console change was an emergency fix worth keeping.

Check yourself

Why does infrastructure-as-code need a state file?To map resources in your configuration to the real objects that exist, by provider identifier. That mapping is not derivable — names are not unique and tags can be edited.
What is the most surprising thing in a state file?Generated secrets, in clear text. It records resource attributes including passwords and connection strings, so it must be protected like a secret store.
What should you review, the code or the plan?The plan. Code says what you intended; the plan says what will happen to production. And search it for 'forces replacement' against anything stateful.
How should you split state files?By blast radius and change cadence — network apart from application. Reference across boundaries with a data lookup rather than shared state, and accept that too many small states creates its own coordination fragility.
← Getting code into production without breaking itDesigning a pipeline people do not route around →