Identity & Access · Chapter 6 of 16
Secrets management
The problem every engineering organisation has, and most solve badly.
Applications need credentials: database passwords, API keys, signing keys, certificates. Where those live is one of the most consequential and least examined decisions in a codebase.
The maturity ladder
| Level | Where the secret lives | The problem |
|---|---|---|
| 0 | Hard-coded in source | In every clone, every fork, and git history forever |
| 1 | In a .env file on the server | Unrotatable in practice; nobody knows who has read it |
| 2 | In CI/CD variables | Better, but broad blast radius and often visible in logs |
| 3 | In a secrets manager, fetched at boot | Good. Still long-lived, still one leak from exposure |
| 4 | Dynamic, short-lived, fetched per use | The target. A leaked credential expires before it is useful |
| 5 | No secret at all — workload identity | The endpoint. The platform attests the workload; nothing to leak |
Levels 4 and 5 are what modern secrets tooling is for. Cloud workload identity — IAM roles for service accounts, managed identities, OIDC federation from CI — removes the secret entirely: the platform proves what the workload is, and the workload receives a short-lived token. The best secret is the one that does not exist.
What to check
- Has the repository ever contained a secret? History matters — rotating without purging history leaves it retrievable. Secret scanning on every push, and on history, is now table stakes.
- Can you rotate without downtime? If not, rotation will not happen after an incident when it matters most. Practise it.
- Who can read each secret, and is that logged? A vault with permissive read policies is a well-organised secret sprawl.
- What is the lease duration? This single number tells you most of what you need to know about the maturity of the setup.
Why secrets management fails in practice
Nearly every organisation has a secrets manager. Most still have secrets in source control, in environment files on developer laptops, and in a shared password vault someone set up in 2019. Deploying the tool is the easy part and it is not the programme.
| Failure | Why it happens | The fix that actually works |
|---|---|---|
| Secrets in git history | Rotating is work; deleting the file feels like enough | Assume any committed secret is compromised and rotate it. Removing it from history does not un-disclose it. Then add pre-commit and server-side scanning so the next one never lands |
| The vault holds them, and they never change | Nobody knows what breaks on rotation | Rotate on a schedule from day one, while the blast radius is small. A credential that has never been rotated cannot be rotated in an emergency |
| One credential shared by five services | It was convenient once | You cannot rotate or revoke it without an outage, so you never do. One identity per workload, always |
| Long-lived cloud keys in CI | It is the default in every tutorial | Short-lived federated credentials. This removes the class rather than protecting it, and is usually a day of work |
| The vault credential itself is a static secret | Something has to bootstrap | This is the actual hard problem — see below. It is solvable and it is where the thinking belongs |
The bootstrapping problem, and the answer
To read a secret from the vault, a workload needs a credential. Where does that come from? If it is a static secret in a config file, you have moved the problem rather than solved it — and moved it somewhere with more access than before.
The answer is an identity the platform can attest to, rather than a secret the workload holds. A cloud instance role, a Kubernetes service account token bound to a specific workload, or a hardware-rooted attestation. The platform vouches for what is running, and the vault trusts the platform.
That is the whole idea behind workload identity, and it is why it matters more than the vault choice. It converts ‘prove you know a secret’ into ‘prove you are the thing you claim to be’, and the second is not stealable by copying a file.
The diagnostic question: “if I copied this container image and ran it on my own laptop, could it read production secrets?” If yes, the secret is in the image or its configuration and the vault is decoration.
And the honest limit: something at the bottom is always trusted — the cloud provider's metadata service, the cluster's signing key, the hardware root. You are not eliminating trust; you are reducing it to one well-defended thing that is hard to copy.
Check yourself