Security Engineering · Chapter 5 of 12
Threat modelling that produces decisions
Forty minutes, a whiteboard, and four requirements in a ticket — not a document nobody reads.
Threat modelling has a reputation for producing large documents and no change. That reputation is earned by how it is usually run, and the version that works is small, fast and repeated. This chapter is that version.
The four questions
Strip away the frameworks and every useful threat model answers four questions in order:
- What are we building? Draw the data flow — on a whiteboard, in ten minutes, badly. Components, the data moving between them, and where it is stored.
- What can go wrong? The systematic part, covered below.
- What are we going to do about it? For each thing: mitigate, transfer, accept, or eliminate. Accept is a legitimate answer and writing it down is the point — an accepted risk with a name on it is a decision, and an unexamined one is a surprise.
- Did we do a good job? Which mostly means: did anything change in the design as a result?
The diagram is the work
The single most useful thing in the whole practice is drawing the trust boundaries — the points where the level of trust in the data or in the caller changes. Between the internet and your edge. Between your edge and your services. Between your application and its data. Between your tenant and another tenant.
Almost every real vulnerability lives at a boundary crossing, because that is where an assumption gets made. Inside a component the code trusts itself; across a boundary it must decide what to trust, and the bugs are in that decision.
The systematic part is asking the same six questions at each crossing — spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege. It is a checklist rather than an insight, and that is precisely its value: it catches the category nobody in the room was thinking about, which is the whole reason to be systematic rather than creative.
How to actually run one, in forty minutes
Two engineers who built it, one person who did not. The outsider asks the questions the builders have stopped seeing. More than four people and it becomes a meeting.
Whiteboard, not a tool. A diagramming tool turns the session into a drawing exercise. Draw it badly and quickly; the diagram is scaffolding, not a deliverable.
Timebox it to forty minutes and accept that you will not be exhaustive. A forty-minute model that produces three changes beats a three-day model that produces a document.
The output is three or four security requirements written into the feature's own ticket — not a separate artefact, not a wiki page. If it lives with the work it gets done; if it lives in a security repository it does not.
And the test: did the design change? If nothing changed, either the design was already sound — possible, and worth saying explicitly — or the session was theatre. Ask which, out loud, at the end.
When to do it, which matters as much as how
- New system or major component: yes, at design time, before code exists. This is where it is cheapest and most valuable.
- New trust boundary: always. A new integration, a new tenant model, a new external caller, a new data store. The boundary is the trigger, and it is a better trigger than a calendar.
- A change in what the system handles: the moment personal data, payment data or credentials enter a system that did not have them, the model is stale.
- Not every sprint. Threat modelling on a cadence produces attendance rather than analysis, and it is how the practice gets a bad name.
The variants worth knowing
| Approach | What it is | When it fits |
|---|---|---|
| The six-category checklist | Ask six threat types at each boundary crossing | The default. Systematic, learnable in an hour, works for almost everything |
| Attack trees | Start from an attacker goal and enumerate paths to it | When you have one specific asset you must protect and want depth rather than breadth |
| The four-question protocol | Just the four questions, no framework | When you have fifteen minutes and no security engineer. Genuinely better than nothing |
| Kill-chain or tactic mapping | Map to a catalogue of real attacker techniques | Detection engineering — it answers 'what would we see', not 'what could go wrong' |
| Privacy-focused variants | Substitute privacy harms for security threats | When the risk is regulatory and reputational rather than technical |
Threat modelling an AI feature, where the boundaries moved
The framework transfers, and two things about the picture change in ways worth naming explicitly:
- A new trust boundary appears where retrieved content enters the prompt. Any document, web page, tool description or previous message that the model reads is untrusted input that will be interpreted as instructions. There is no parameterised channel to separate them, so this boundary cannot be secured at the input layer — only by constraining what the model is permitted to do downstream.
- The model's output crosses a boundary too. If output reaches a tool, a shell, a database or a browser, it is untrusted input to that system. Treat it exactly as you would treat a user-supplied string, because functionally that is what it is.
- And elevation of privilege takes a new form: the agent has the union of every permission it was given, and an attacker who can influence its input inherits that union. So the meaningful control is scoping the agent's authority per task, not filtering its prompt.
- The rest of the checklist applies unchanged, which is the reassuring part — the discipline is not new, the diagram just has an extra boundary on it.
Check yourself