Security Engineering · Chapter 6 of 12
Two threat models, worked
The practice from the previous chapter applied to a retrieval chatbot and a refund API — the diagram, the boundaries, and the tickets that came out.
The previous chapter is the method: four questions, a bad whiteboard drawing, trust boundaries, forty minutes. This one runs it twice, on systems you might actually be asked to build — because the method reads as obvious and is surprisingly hard to do for the first time without an example.
Example 1 — a chatbot over internal documents
The system: staff ask a question, the app searches an index of company documents, pastes the best passages into a prompt, and a model answers with citations.
Why TB3 is the one that gets missed
A trust boundary is where the level of trust in the data or the caller changes. Everyone draws it at the internet edge, because that is where the untrusted things obviously are.
But content retrieved from your own index crosses into a channel where it is read as instructions. Trust went from ‘a document someone wrote’ to ‘text the model will act on’. That is a change in trust level, so it is a boundary, and it sits entirely inside your own network.
The tell that you missed it: your diagram has one dashed line, at the left edge. Real systems usually have three or four, and the interesting ones are internal.
Walking the six questions at each boundary — spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege — most answers at TB1 are the ordinary web ones and take a minute. The value concentrates at TB2 and TB3.
Note what the exercise did NOT produce: a risk score, a matrix, or a document. It produced four tickets, two of which stop the launch. The team can act on that this afternoon — and the two that do not block are still recorded, so nobody rediscovers them in an audit.
Example 2 — a refund API
The system: a support agent (a human) opens a case in the support tool, clicks Refund, and an internal API moves money back to the customer's card via a payment provider.
Same method, and the boundaries fall in different places — which is the point of doing a second one:
| Boundary | What changes | What you find there |
|---|---|---|
| Support tool → refund API | A human's intent becomes a machine call | Does the API trust the tool, or the person? If the tool is trusted, anyone who can reach the API can refund without being an agent. |
| Refund API → ledger | A request becomes a financial fact | Idempotency. A retried request that refunds twice is not a security bug until someone notices it can be triggered deliberately. |
| Refund API → payment provider | Money leaves your control | Their outage is your inconsistent state. What happens if they succeed and your write fails? |
| Anywhere → the amount field | Untrusted number becomes an instruction | Server-side bound against the original transaction. Never trust a client-supplied amount, and never trust one you did not re-read from your own ledger. |
And the finding that a technical threat model reliably surfaces and a technical review does not:
Can the same person raise and approve a refund?
It is not a coding flaw. Every component works exactly as specified. It is a segregation of duties failure, and it is the shape of most insider fraud — no exploit, no intrusion, just an authorised person doing an authorised thing twice.
You find it by asking the elevation-of-privilege question at the first boundary and taking it seriously, rather than answering ‘the agent is authenticated, so fine’. Authenticated is not the same as authorised, and authorised is not the same as authorised alone. See risk and controls, where this is a named control failure with a standard test.
What the two examples have in common
- The interesting boundaries were internal. In both cases the edge was the least interesting line on the diagram, and the findings sat between components the team owned.
- The best finding in each was not a vulnerability. One was a missing permission filter; the other was a missing second approver. Neither would appear in a scanner, a penetration test, or a code review — which is precisely the gap this practice fills.
- Both took under an hour, and both produced launch-blocking work that would otherwise have been found later and more expensively.
- Half the tickets were logging and deletion. Threat modelling produces unglamorous, cheap, high-value work — and the reason it keeps getting cancelled is that the output does not look impressive next to a penetration test report.
Running your first one
- Pick something being built now, not something in production. The output has to be able to change the design or the session is theatre.
- Draw the diagram in the room, badly. Someone will correct it within two minutes, and that correction is usually itself a finding — the diagram people carry in their heads is not the same one.
- Timebox to forty minutes and take the boundaries in order. Do not let the session become a design review.
- Write the accepted risks down with a name against each. That is the row people skip, and it is the one that protects the team later.
- Judge it on one question only: did anything change as a result? If nothing did, either the design was already sound — possible — or the session was a meeting.
Check yourself