Failure and incident narrative

Handbook · Chapter 6

Failure and incident narrative

Every archetype asks for this story. The outage is the setting; the decisions are the content.

There is no loop on this site without a version of “tell me about something that broke”. Candidates over-invest in the technical diagnosis — which the interviewer cannot verify and does not much care about — and under-invest in the decisions, which are the entire point.

Four moments carry the score, and none of them is the root cause. Why detection took as long as it did, who decided to escalate, why you mitigated before understanding, and which action item actually shipped. Rehearse those four answers for one real incident and you have prepared for this question everywhere it appears.

The structure that works

Impact first, in user terms. “Checkout failed for about 8% of users for 31 minutes” — not “the pod was OOMKilled”. Leading with the mechanism signals you think the system is the point.
Detection, honestly. If a customer told you before your monitoring did, say so. It is the most common real answer and pretending otherwise is transparent.
The decision under uncertainty. The centre of the story: what you did before you understood the cause, and why. Rolling back without knowing why is usually correct and candidates apologise for it — do not.
Root cause, briefly. Two sentences. This is the part you want to spend ten minutes on and should spend one.
What changed afterwards, and whether it shipped. The strongest possible ending is an action item that landed, with a date. The second strongest is an honest “it is still in the backlog and here is why”.

Blameless, when the person is in the room

Now a standard question, and it is not asking about your feelings. It is asking whether you can hold two things at once: the outcome was bad, and the person is not the cause. The answer that lands moves the frame from the individual to the system that let one person do that much damage — “the question isn't why Priya ran it, it's why a single command could drop a production table with no confirmation and no backup check”.

The trap. Saying “we're blameless” and then describing the incident in terms of what someone failed to do. Interviewers listen for this specifically, and the contradiction is very audible.

The other trap. Blamelessness used to avoid accountability entirely. A good answer still ends with someone owning the fix.

If you have never been on-call

Use what you have, and say what it is. A build you broke for the team, a migration that corrupted data in staging, a bug you shipped and caught. The structure is identical and the honesty is worth more than a borrowed story. What does not work is claiming incident-command experience you do not have — the follow-up questions in this chapter's diagram will find it within two minutes.

The drill

Pick one real incident. Write the four answers from the diagram, one paragraph each.
Time yourself telling it. Target three minutes; most people run to eight because of the root-cause section.
Then prepare a second story where you caused it. Being asked for that one and not having it is a visible gap.

← Discovery before solution Reading a job description →