Security Engineering · Chapter 9 of 12
Incident response and threat hunting
What actually happens during an incident, the containment decision nobody wants to make live, and how hunting differs from waiting for an alert.
The detection chapter is about building the alarms. This one is about the night one goes off — and about looking for what never set one off at all.
An incident is not an alert
An event is anything that happened. An alert is an event a rule thought was interesting. An incident is an alert a human has confirmed matters. The ratios between those three are brutal, and they are why the first-line analyst job is what it is: a large security operations centre may process millions of events into thousands of alerts into a handful of incidents.
Which makes triage the highest-volume skill in security, and the one most affected by alert quality. An alert that fires into a queue nobody reads is not a detection — it is a liability, because it will be in the timeline afterwards, unread.
Containment: the decision that defines the incident
Isolate now, or watch first?
Isolate immediately. You stop lateral movement. You also lose memory-resident evidence the moment you pull the plug, and you tell the attacker they have been spotted — at which point a competent one may burn what they have and detonate whatever they were saving.
Watch and map. You learn the full extent of their access, which is the only way to be confident eradication is complete. Meanwhile they may be exfiltrating.
There is no general answer — and that is the point. The answer depends on what is at stake, and it should be decided in advance, in writing, by someone senior enough to own it.
The single most valuable artefact in an IR programme is a short document saying who may authorise disconnecting a production system, and under what conditions. Without it, that conversation happens at 3am with people who have never met.
And the partial-containment trap. Removing the attacker from three of four footholds is worse than removing none: you have taught them you are looking, without removing them. Containment is a single co-ordinated action, planned then executed — not a series of fixes as you find things.
Eradication and the question people skip
Eradication is not deleting the malware. It is removing the access — which means answering: how did they get in, what did they create, and what did they take?
- Every credential in scope is burned, including service accounts, API keys and anything in memory on a compromised host. A credential that was present on a compromised machine must be rotated whether or not you can prove it was stolen.
- Persistence is the part that gets missed. Scheduled tasks, new service accounts, an added SSH key, an OAuth grant, a mailbox forwarding rule, a modified pipeline. Attackers create several routes back precisely because they expect you to find one.
- The initial access vector must be closed, or you are rebuilding into the same hole.
- Rebuild rather than clean, where you can. Trusting a cleaned host is a judgement about whether you found everything, made by someone who has just been demonstrated not to have found everything.
Threat hunting — the inverse of detection
Detection waits for a rule to fire. Hunting assumes the rules did not. It is a deliberate search through your own telemetry for activity that is happening now and has produced no alert.
The framing that makes it click: a hunt starts from a hypothesis, not from an alert. ‘If an attacker were using a legitimate remote-access tool to move between our servers, what would that look like in the logs, and is it there?’ You go and look. Most hunts find nothing, and that is a normal outcome rather than a failure — though a hunt that finds nothing should still produce something.
| A hunt produces | Which is worth |
|---|---|
| A finding — something real was happening | Obviously. But this is the rarest outcome. |
| A new detection rule | The most common useful output. You looked manually, you now automate it. |
| A visibility gap | “We could not answer that question” is a finding about your logging, and often the most valuable one. |
| A tuned rule | You found the benign cause of a noisy alert and can now suppress it honestly. |
MITRE ATT&CK is the shared vocabulary for this. It is a catalogue of the techniques attackers actually use, observed in real intrusions, organised by tactic — initial access, persistence, privilege escalation, lateral movement, exfiltration. Its practical value is not the taxonomy; it is that it turns ‘are we covered?’ from an opinion into a map you can colour in. Which techniques can you detect, which can you only detect sometimes, and which would you miss entirely?
The honest caution about ATT&CK coverage maps.
A coloured grid showing 80% coverage is a persuasive artefact and a misleading one. “We have a rule” is not the same as “we would catch it” — the rule may be too narrow, may fire into an unread queue, or may depend on a log source that stopped shipping three weeks ago.
The version of that exercise worth doing is adversarial: have someone actually perform the technique and see whether anybody notices. That is what purple teaming is, and it converts a claim into a measurement.
Preparation, which is the only phase you control calmly
- A contact list that works out of hours, including legal, communications, and whoever can authorise the containment decision above. Kept somewhere that survives the systems being down — an incident plan stored only on the compromised intranet is a recurring joke because it keeps happening.
- Retention long enough to investigate. Dwell times run to months; logs kept for 30 days answer nothing. This is the single most common reason an investigation cannot establish scope, and it is a budget decision made long before the incident.
- Know your reporting clocks before you need them. Regulators impose deadlines measured in hours, and they start from awareness, not from resolution. Find out which apply to you now.
- Tabletop exercises — walk through a scenario in a room. They reliably surface the unglamorous gaps: nobody knows who calls the insurer, the on-call rota has a person who left, the runbook assumes access to a tool that requires the single sign-on that is down.
- Decide the ransom position in advance, with legal. Not during.
The post-incident review, and how to not waste it
Blameless, and specific. The output is not a list of people who should have been more careful; it is a small number of changes with owners and dates. The same discipline as an outage post-mortem — see reliability — and the failure mode is identical: a document nobody reads and no change that ships.
The two questions worth more than the rest of the review: how long was the attacker in before we noticed, and what would have made that shorter? Everything else is downstream of the answer.
Check yourself