Security Engineering · Chapter 4 of 12
Detection engineering, and responding to what you find
Writing detections as code, tuning without going blind, and the first hour of an incident.
Detection is a signal-processing problem with a legal consequence attached. Too sensitive and your analysts stop reading; too specific and you catch only the attacker who behaves exactly as you predicted. This chapter is about running that trade-off deliberately rather than by accident.
What to collect, in priority order
You cannot detect what you do not log, and log volume costs real money, so the ordering matters.
| Source | What it catches | Note |
|---|---|---|
| Identity and authentication | Credential stuffing, impossible travel, MFA fatigue, new-device sign-in | The highest-value source by a wide margin. Identity is the perimeter, so it is where intrusions begin |
| Cloud control plane | Role creation, policy change, key creation, logging disabled | Small volume, extremely high value. ‘Audit logging turned off’ should page |
| Endpoint (EDR) | Process execution, persistence, credential access | Rich and noisy; needs real tuning investment |
| DNS and egress | Command-and-control, exfiltration, newly registered domains | Cheap, and catches things nothing else does |
| Application logs | Business-logic abuse, enumeration, bulk export | The most neglected source. Nothing else can see ‘one account exported 40,000 records’ |
That last row is worth emphasising because it is where security and engineering meet. No security product knows what a normal amount of data for one of your users to download is. Only your application does, and a single well-chosen application-level alert frequently outperforms an expensive detection platform for the abuse cases that actually matter to your business.
Detection as code
A detection is software: it has a version, a test, a review and an owner. Treating detections as configuration typed into a console is why so many organisations cannot answer ‘why does this rule exist and who changed it?’
- In version control, in a structured format such as Sigma, so they are portable across platforms and reviewable in a pull request.
- With a test — a sample of the malicious behaviour that must fire, and a sample of the benign lookalike that must not. Without the second one you have no protection against a rule that fires on everything.
- With documented context: what it detects, why it matters, the expected false positive rate, and what the analyst should do. A detection without a response is an alarm with no one assigned to it.
- Mapped to a framework — MITRE ATT&CK technique IDs — so you can answer ‘what are we blind to?’ by looking at coverage rather than by intuition. The honest use of ATT&CK is as a gap map, not as a scoreboard; a heat map with everything green usually means the coverage claims are generous.
Tuning: never move a threshold without a labelled set
The instinct when a rule fires 400 times a day is to raise the threshold. That reduces the number and tells you nothing about whether you have just tuned away the thing you built it for.
Label first. Take a few months of alerts and classify them: genuine, explainable, unclear. That labelling is the only thing that lets you say afterwards whether a change was an improvement rather than a reduction.
Then look at what the false positives have in common. They will cluster — a backup agent, a vulnerability scanner, one team's deployment tooling. Those want an exclusion scoped to that identity and that behaviour, not a global threshold change.
Then improve the signal rather than the threshold. If the rule measures the wrong quantity, no threshold will fix it — adding one more discriminating feature usually removes most of the noise without losing sensitivity.
And write down the trade-off. If you cut false positives by 80%, what is the estimated effect on detection? In two years someone will ask why the threshold is where it is, and ‘it reduced alert volume’ is not an answer that survives an audit.
The first hour of an incident
The order matters, and the most common failure is doing step three before step two.
- 1. Verify. Is this real? A large share of escalations are misconfiguration or an unannounced test. Verifying costs minutes; a full response costs days.
- 2. Scope, before you contain. Which identities, which systems, since when. The temptation is to kill the compromised host immediately, and doing so before you have scoped destroys the evidence of how they got in and where else they went — and you will find them again next week.
- 3. Preserve. Snapshot volumes and memory before changing anything. Memory in particular is gone forever the moment you power off, and it holds the things disk does not.
- 4. Contain. Isolate at the network layer rather than powering off, so the system stays available for analysis. Revoke sessions and rotate credentials — note that disabling an account does not invalidate an already-issued token, which is a gap people discover during the incident rather than before it.
- 5. Eradicate, then recover. And recovery means rebuilding from a known-good image, not cleaning. You cannot prove a cleaned host is clean.
- 6. Learn. A blameless postmortem with a timeline, the detection gap and the control gap. If the detection came from a customer rather than from you, that is the most important finding in the document.
Purple teaming, which is the cheapest maturity you can buy
Red team finds a way in. Blue team defends. Purple teaming is the two working together: the red side executes a specific technique, deliberately and announced, and the blue side checks whether it was logged, whether it alerted, and how long it took to notice.
The output is a concrete gap list — ‘we log this and do not alert on it’, ‘we do not log this at all’ — and it is far more valuable per pound than an annual penetration test, because the point is not to find out whether you can be compromised. You can. The point is to find out what you would see.
It is also the honest answer to “how do you know your detections work?” A rule that has never fired might be perfect coverage of something that never happened, or it might be broken. Only executing the technique tells you which.
Check yourself