Practice
Answer out loud first
Every question below is drawn from a real loop in the atlas. Say your answer aloud before you reveal anything — reading a good answer teaches you far less than discovering the gap between it and yours. Each reveal also shows what the question is testing, which is usually not what it appears to ask.
A customer says the model is wrong 30% of the time. What do you do first?
A strong answer
Ask what “wrong” means and how it was measured, before proposing anything. Who labelled those cases? Was it 30% of all traffic or 30% of the ones they noticed? Is there a written definition of correct? Very often no measurement exists at all, which makes your first deliverable an evaluation harness rather than a model change.
What is being tested
Whether you find the problem before solving it. Proposing fine-tuning here is the single most common way to fail this round.
A customer wants to migrate off a competitor. Walk me through your first meeting.
A strong answer
Discovery, not a pitch. What are they running now and what does it cost them; what specifically triggered this conversation; who has to sign off; what would make them stay. Then map current state to target state and be explicit about what will be hard — naming a real migration risk early is what builds technical credibility.
What is being tested
Whether you can resist demoing. Interviewers score the first ten minutes.
Three teams refuse to migrate to your platform. What do you do?
A strong answer
Treat it as a product problem, not a compliance problem. Find out what the platform costs them that their current path does not — usually a missing capability, a migration cost nobody funded, or a reliability reputation. Fix the reason or fund the migration. Mandate is the last resort and it tells you the platform was not worth adopting.
What is being tested
Whether you run platform as a product with customers who can say no.
You have two weeks and an executive demo. What do you cut?
A strong answer
Name the one workflow that must work end to end, and cut everything else explicitly — languages, edge cases, integrations, polish. Say what you are cutting and why, and what you would need to add it back. A demo that does one thing convincingly beats one that does five things partially.
What is being tested
Scoping. It is a scoping test wearing a delivery costume.
How do you evaluate a multi-turn agent, where there is no single correct output?
A strong answer
Score the task outcome rather than the text: did the user's goal get achieved, in how many turns, with how many tool errors. Layer on a rubric with pairwise comparison rather than absolute scoring, since absolute scores compress. Calibrate any LLM judge against human labels on a subset and report agreement rate, not just the score. Be explicit about what you cannot measure this way.
What is being tested
Whether you have actually built an eval or only read about them.
You changed a prompt and the demo looks better. Do you ship it?
A strong answer
No — a demo is a sample of one. Run the golden set, compare to the recorded baseline, and check the delta against the noise floor for that set size. On 200 cases a three-point move is noise. Also check the regression suite, because prompt changes routinely fix one class of failure and open another.
What is being tested
Whether you ship on evidence or on vibes. This is the fastest disqualifier in agent interviews.
Your LLM judge agrees with humans 85% of the time. Is that good enough to ship on?
A strong answer
It depends entirely on where the 15% falls. If disagreement is uniform and random, an 85% judge is usable for tracking trends. If it concentrates on the cases that matter — safety refusals, high-value customers, the failure mode you are trying to fix — it is worse than useless because it will be confidently wrong exactly where you need it. Ask for the error distribution before answering.
What is being tested
Whether you interrogate an aggregate before trusting it.
How would you measure developer productivity without gaming it?
A strong answer
Accept upfront that every proxy is gameable, and say so — that is the answer's opening. Then pair a quantitative signal (time from commit to production, CI flake rate, build time p95) with a qualitative one (a recurring developer survey), because the survey catches what the metric misses and vice versa. Never measure individuals; measure the system.
What is being tested
Whether you know that the naive answer — lines, commits, story points — is a trap.
Reliability improved this quarter. How do you know it was you?
A strong answer
Compare like for like. Traffic mix, release cadence and dependency behaviour all move independently of your work, and a quiet quarter looks identical to a good one. Anchor on a change you made, and find the period before and after with comparable load. If you cannot separate them, say so.
What is being tested
The control habit, in SRE vocabulary.
Your eval improved by two points. Convince me that is real.
A strong answer
Multiple seeds, and report variance rather than the best run. Check the set size against the effect — two points on a small held-out set is inside the noise. Rule out contamination between train and eval. Then show the ablation: which single change produced it, and does removing that change give the points back.
What is being tested
Empirical rigour, and whether you know your own result's failure modes.
How would you know this deployment succeeded, six months from now?
A strong answer
Define the success measure before the engagement starts and get it agreed in writing. Then name the comparison: succeeded relative to what they were doing before, measured on the same population. Usage alone is not success — a tool people open and abandon shows usage.
What is being tested
Whether you think in outcomes or in shipped features.
Offline AUC improved and the A/B was flat. Give me your hypotheses, in order.
A strong answer
Training-serving skew — features computed differently in the two paths. Then the offline set is not representative of live traffic. Then position and exposure bias: your training labels came from what the old model already showed users. Then the metric mismatch — AUC moved but the business metric depends on the top few ranks. Then insufficient power: the A/B may not have been able to detect the effect size.
What is being tested
The signature question of the archetype. An ordered list beats a longer unordered one.
Justify your team's headcount to a CFO.
A strong answer
Engineer-hours recovered per week times loaded cost, against team cost. Then the harder half: name what you are not counting and what would falsify the claim. A CFO trusts a number that comes with its own caveats far more than a clean one.
What is being tested
Whether you can argue in business terms rather than engineering ones.
Design a service handling 50k QPS at 99.9% availability.
A strong answer
Work the ladder out loud with numbers. Peak factor over average. Measured or assumed per-node capacity at a stated utilisation. Raw node count. Then multiply for failure domain — n+1 per availability zone across three zones — and again for growth headroom. End with a machine count and your assumption list. Then name the binding constraint: memory, IOPS, network, or a downstream rate limit.
What is being tested
NALSD. Non-abstract is the exam — a component diagram with no arithmetic fails.
Estimate the GPU memory to serve a 70B model at batch 32 with 8k context.
A strong answer
Decompose out loud. Weights: 70B parameters at fp16 is roughly 140GB, so this is multi-GPU before you start. KV cache: scales with batch times context times layers times hidden size times two for K and V — at batch 32 and 8k context this is tens of gigabytes and is the term candidates forget. Add activation and framework overhead. Then say what you would do about it: quantise the weights, page the cache, or shard across more devices.
What is being tested
Whether you reach for the KV cache unprompted. Most candidates give only the weights.
Design multi-tenant isolation for untrusted workloads.
A strong answer
Namespaces are the wrong first answer — they are an organisational boundary, not a security one. The real boundary is the kernel: separate nodes per tenant, or a sandboxed runtime such as gVisor or Firecracker, or full VM isolation depending on the threat model. Then layer network policy, resource quotas to prevent noisy-neighbour denial of service, and separate credentials per tenant.
What is being tested
Whether you know where the actual trust boundary is.
Your Terraform apply would have deleted a production database. How did your process allow that?
A strong answer
Plan output was not reviewed, or was reviewed by someone who could not read it. Fixes: require plan review with destructive changes highlighted, use lifecycle prevent_destroy on stateful resources, separate state files so blast radius is bounded, and run applies through a pipeline with an approval gate rather than from a laptop.
What is being tested
Whether you think in blast radius. Blaming the person fails this.
Your agent's p50 is fine but users say it is slow. Where do you look?
A strong answer
Instrument the whole turn, not each call. If a turn chains four calls that each have a 1% tail, roughly 3.9% of turns hit at least one tail — so users experience worst-case latency about four times more often than any per-call dashboard suggests. Then look for sequential calls that could run in parallel, and timeouts set above the downstream p99.
What is being tested
The one genuinely counterintuitive result in agent engineering.
Cut inference cost 40% without hurting quality. What do you try, in what order?
A strong answer
Cheapest first. Cache repeated calls. Trim the context you resend every turn. Route easy requests to a smaller model and keep the large one for the hard tail. Then batch. Then quantise. Then, only if needed, distil. Measure quality against your golden set after each step, and stop when the eval moves — the ordering matters more than the list.
What is being tested
Whether you sequence by cost-to-try, and whether you gate each step on an eval.
Your observability bill is larger than your compute bill. What do you cut first?
A strong answer
Find the cardinality. One high-cardinality label — user ID, request ID, full URL — on a metric multiplies stored series by the number of distinct values, and that is almost always the cause. Move that data from metrics into traces, tail-sample traces so you keep the slow ones, and keep aggregate metrics low-cardinality. Then say which queries you are giving up, because a cut with no cost is one nobody believes.
What is being tested
Whether you understand the mechanism rather than just proposing to retain less.
How would you price a feature whose cost varies per use?
A strong answer
Start from cost per resolved task, not per API call — retries and failures are real cost. Model the distribution, not the mean, because a small tail of expensive sessions can invert unit economics. Then choose a structure that bounds your downside: usage tiers, a fair-use cap, or seat pricing with a usage ceiling.
What is being tested
Whether you know the denominator is the whole question.
Walk me through an incident you were part of.
A strong answer
Impact first in user terms and duration. How you found out, honestly — including if a customer told you first. Then the centre of the story: what you did before you understood the cause, and why. Root cause in two sentences. End with the action item that actually shipped, with a date, or an honest account of why it has not.
What is being tested
The decisions, not the diagnosis. Spending eight minutes on root cause is the common error.
How would you run a blameless postmortem with the person who caused the outage in the room?
A strong answer
Move the frame from the person to the system. The useful question is not why they ran the command, but why a single command could do that much damage with no confirmation, no dry run and no backup check. Ask what information they had at the time — usually their decision was reasonable given what they could see, and that gap is the finding. End with an owner for the fix; blameless is not accountability-free.
What is being tested
Whether you can hold ‘bad outcome’ and ‘not this person's fault’ at once.
Your error budget is exhausted in week two of the quarter. Product wants to ship. What happens?
A strong answer
The policy should already exist and should have been agreed when nobody was under pressure — that is the point of writing it down. Typically: feature freeze until the budget recovers, with a named exception path and an escalation owner. If no policy exists, say that, and say that writing one is the actual deliverable. Then be honest that this is a negotiation and describe how you would run it.
What is being tested
The hardest SRE round. There is no clean answer; they want your reasoning and escalation path.
Your agent calls a tool that fails silently and returns an empty result. How do you detect it?
A strong answer
Not with a better prompt. Validate tool responses against a schema and treat empty as an error rather than a value. Trace every tool call with status and latency. Add an eval case that exercises exactly this failure so a regression catches it. Then decide the behaviour deliberately: retry, fall back, or surface the failure to the user — silently continuing is the worst option.
What is being tested
Whether you reach for observability and evals rather than prompt engineering.
The customer's data is far worse than they told us. Walk me through that conversation.
A strong answer
Lead with what it means for their timeline, not with the deficiency. Bring the revised plan to the same meeting — what is still achievable, what slips, what you need from them. Do not sugar-coat and do not blame; the relationship survives bad news delivered early far better than good news that turns out to be wrong.
What is being tested
Whether you can deliver bad news without either softening it into uselessness or assigning fault.
What would you deprecate in your first ninety days, and how would you know you were wrong?
A strong answer
Name something specific and unglamorous, with a migration path and a date. Then the second half, which most candidates skip: the signal that would tell you to stop — a team blocked with no alternative, a use case you did not know about, a cost that lands on someone else's budget. Deprecation without a reversal criterion is how platform teams lose trust.
What is being tested
Whether you can kill something you did not build, and admit being wrong about it.
What is the strongest argument against your own last result?
A strong answer
Have a real one. Contamination, a baseline you did not tune as hard as your method, an effect that only appears at one scale, a metric that rewards your approach's failure mode. Naming the genuine weakness is a strength signal; claiming there isn't one is the weakest available answer.
What is being tested
Intellectual honesty, and whether you have actually stress-tested your own work.
Tell me about a deal you lost.
A strong answer
Give the specific technical reason. “We lost on price” is weak; “their existing footprint made migration cost exceed the three-year saving, and I said so before we invested more of their time” is strong. Then what you changed afterwards.
What is being tested
Whether you have real deal experience and can be honest about losing.
Coming: spoken practice
Voice input and read-back are on the roadmap, so you can rehearse aloud and hear the strong answer rather than reading it. The drills throughout the handbook assume you are speaking, because every decisive round on this site is spoken.