Practice
Answer out loud first
Every question is drawn from a real loop. Say your answer aloud before revealing anything — reading a good answer teaches you far less than discovering the gap between it and yours. Each reveal gives a full answer plus what the question is testing, which is usually not what it appears to ask.
A customer says the model is wrong 30% of the time. What do you do first?
What you said
A strong answer
I would not propose anything yet, and I would say so out loud, because the fastest way to fail this is to start solving. My first questions are about the number itself. Who measured it, on what sample, and against what definition of correct? Is it 30% of all traffic, or 30% of the cases someone happened to notice and escalate — because those differ by an order of magnitude and only the second one is usually true. Is there a written definition of a correct answer, or is each reviewer applying their own judgement? Has anyone looked at whether the errors cluster — one document type, one customer segment, one question shape?
In my experience the answer to ‘how did you measure it’ is very often ‘we didn't, really’ — someone senior saw three bad outputs in a demo and extrapolated. If that is the case, then the real first deliverable is not a model change at all. It is an evaluation: fifty to a hundred labelled cases drawn from real traffic, a written definition of correct, and a baseline score. Without that we cannot tell whether any fix helped, and we will end up shipping changes and arguing about vibes.
Once I have that, the diagnosis is usually mundane. Most ‘the model is wrong’ problems turn out to be retrieval problems — the right document was never fetched, so the model was answering from nothing. That is a chunking, indexing or permissions issue, and it is fixable in days rather than weeks. Only after ruling that out would I look at prompting, and only after that at the model.
What is being tested
Whether you establish what ‘wrong’ means before proposing a fix. Reaching for fine-tuning here is the single most common way to fail this round, and interviewers count how many minutes pass before you propose something.
A customer wants to migrate off a competitor. Walk me through your first meeting.
What you said
A strong answer
The first meeting is discovery, not a pitch, and I would go in having decided not to demo anything unless asked.
What I want to leave with: what they are running today and how it is actually used, not the architecture diagram; what specifically triggered this conversation, because ‘we are evaluating options’ usually hides a renewal date, a price rise, an outage or a new person with a mandate; who has to agree, including the people not in the room — security review, procurement, the team who will do the migration work; what they have already tried; and what happens if they do nothing, because sometimes the honest answer is that doing nothing is fine and I would rather know in week one.
Then I would map current state to target state out loud and name the hard part. There is always one — a proprietary feature with no equivalent, a data migration nobody has scoped, three years of operational tooling built around the incumbent. Naming it in the first meeting is counterintuitive and it is the single fastest way to build technical credibility, because they already know it exists and every other vendor has skipped past it.
Finally I would agree what a successful proof of concept would prove, in writing, before we build one. POCs without written success criteria never end; they just gradually stop.
What is being tested
Whether you can resist demoing. Interviewers score the first ten minutes.
Three teams refuse to migrate to your platform. What do you do?
What you said
A strong answer
I would treat it as a product problem rather than a compliance one, because a mandate here is an admission that the platform is not worth adopting on its merits.
First, find out why, individually, because the three reasons are usually different. In my experience they cluster into: a genuine capability gap — the platform does not do something they actually need; an unfunded migration — they agree it is better and nobody has given them the six weeks; a trust problem — they were burned by an earlier platform promise; or the platform is genuinely worse for their case, which happens and is worth knowing.
The responses differ. Capability gap: build it, or accept they are a legitimate exception and stop counting them as a failure. Unfunded: do the migration for them — write the code, open the pull requests, ask only for a review. That is the single highest-leverage thing a platform team can do and it is where most migrations succeed or die. Trust: pick the smallest possible win and deliver it exactly as promised.
What I would not do is escalate first. It works once, costs you the relationship, and guarantees the next migration is harder because every team now knows the pattern.
And I would take it as feedback on the platform's adoption metric. If three teams are refusing, more are complying reluctantly, and that is the real number I should be worried about.
What is being tested
Whether you run platform as a product with customers who can say no.
You have two weeks and an executive demo. What do you cut?
What you said
A strong answer
I would name the one workflow that must work end to end, and cut everything else explicitly and out loud — because the failure mode here is not cutting too much, it is cutting nothing and demoing five things that each half-work.
Concretely: pick the single highest-value path, ideally one the executive has personally complained about. Make that path excellent on real data, including the ugly cases. Then cut, by name: other languages, edge cases, the admin interface, error handling beyond a graceful message, and anything requiring a new integration. I would write that list down and share it, so nobody discovers at the demo that something is missing.
I would also decide what happens when it fails in front of the executive, because at two weeks it will. A visible, honest fallback — ‘I'm not confident enough to answer that, here is who can help’ — reads as engineering maturity. Silence or a confident wrong answer reads as a broken product, and that impression is very hard to recover.
And I would agree the demo script with the sponsor beforehand. That is not gaming it; it is making sure the ten minutes are spent on the thing we built rather than on an unrehearsed tangent. If they want to go off script, fine — but I want to know in advance whether that is expected, because it changes what I build.
What is being tested
Scoping, wearing a delivery costume. Strong candidates cut by name and plan for the failure in the room; weak ones promise all five workflows and arrive with none finished.
How do you evaluate a multi-turn agent, where there is no single correct output?
What you said
A strong answer
I would start by rejecting the framing that we score the text, because with multi-turn there is no reference string to compare against and any attempt to build one produces an eval that punishes correct-but-different answers.
What I score instead is task outcome. Define what the user was trying to achieve, and measure whether it happened: was the ticket resolved, was the booking made, did the user have to escalate. That is objective, it is the thing the business cares about, and it survives the model phrasing things differently. Alongside it I track efficiency — turns to completion, tool calls per task, cost per resolved task — because an agent that gets there in fifteen turns is failing even when the outcome is right.
Where outcome is genuinely ambiguous I use pairwise comparison rather than absolute scoring: show a judge two trajectories and ask which is better. Absolute one-to-ten scores compress into seven-to-nine and carry almost no information, whereas pairwise is stable and produces a ranking you can act on. If I use a model as the judge I calibrate it against human labels on a subset and report agreement rate, not just the score.
And I would build a trajectory eval, not just an endpoint one — recorded conversations replayed against the new version, checking whether it takes the same critical actions. That is what catches the change that fixes one class of failure and quietly opens another.
Finally I would be explicit about what none of this measures: tone, whether the user felt helped, and the long tail of situations nobody thought to record. Those need human review on a sample, and I would budget for it rather than pretend the harness covers everything.
What is being tested
Whether you have actually built an eval or only read about them.
You changed a prompt and the demo looks better. Do you ship it?
What you said
A strong answer
No, and not because I doubt it improved — because a demo is a sample of one and prompt changes are notorious for fixing the case in front of you while breaking two you are not looking at.
What I would do takes about twenty minutes if the harness exists. Run the golden set against both versions. Compare to the recorded baseline. Check whether the delta clears the noise floor for that set size — on two hundred cases a three-point move is not a result, and saying so out loud is the discipline that matters. Then look specifically at the cases that got worse, because prompt edits trade off, and the trade is usually invisible in aggregate: overall score flat, but the model now refuses a category it used to handle.
If the harness does not exist, then the honest answer is that we cannot know, and building it is the actual work — which is a much better thing to say in an interview than pretending we can judge from a demo.
Two more things I would insist on. Version the prompt like code, in the repository, with the eval result attached to the commit, so six months later we can answer why it says what it says. And keep a small set of adversarial cases in the suite permanently, because the prompt change that looks harmless is usually the one that weakens a guardrail.
What is being tested
Whether you ship on evidence or on vibes. This is the fastest disqualifier in agent interviews.
Your LLM judge agrees with humans 85% of the time. Is that good enough to ship on?
What you said
A strong answer
It depends entirely on where the fifteen percent falls, and I would not answer until I had seen the error distribution.
If disagreement is uniform and unbiased — the judge is a bit noisy but not systematically wrong — then 85% is perfectly usable for tracking a trend between releases, because the noise averages out over a few hundred cases and what I care about is the delta, not the absolute.
If disagreement concentrates, it is worse than useless, because it will be confidently wrong exactly where it matters. The patterns I would look for: does it disagree more on refusals and safety cases, on longer answers, on a particular customer segment, or on the specific failure mode we are trying to fix? A judge that agrees on the easy 85% and fails on the hard 15% will tell us we improved every time we regress on the thing we care about.
I would also check the direction of the error. A judge that is systematically generous is dangerous in a way a systematically harsh one is not — the harsh one costs us some good releases, the generous one ships bad ones.
And I would ask how the 85% was established. Against how many human labels, by how many labellers, and what was the inter-annotator agreement? If two humans only agree with each other 80% of the time, an 85% judge is at the ceiling and the real problem is that the task definition is ambiguous.
What is being tested
Whether you interrogate an aggregate before trusting it.
How would you measure developer productivity without gaming it?
What you said
A strong answer
I'd open by saying that every proxy is gameable, because that framing is the answer's foundation and interviewers listen for it. Lines of code, commits and story points all measure activity and reward the wrong behaviour; anything measured per-individual becomes a target and stops being a measurement.
So I'd measure the system, not the people, and pair two kinds of signal. Quantitative: time from commit to production, change failure rate, CI flake rate, build time p95, and time-to-first-green for a new joiner. Those are system properties — no individual can inflate them by working differently, and they get better only if the platform gets better. Qualitative: a short recurring developer survey. This is not a soft supplement; it catches what the metrics cannot, like a build that is fast but a local environment that takes a day to set up, and it is what DevEx research consistently finds to be the strongest signal.
Then the discipline: I'd report at team level and above, never per-person; I'd pair every metric with a counter-metric so speed cannot be bought with quality; and I'd expect to change the metric set as the bottleneck moves, because a metric that has been flat for a year is measuring something that is no longer the problem.
And I'd be honest that the point is not the dashboard. The point is to find the next bottleneck, fix it, and prove the fix — the number exists to make the argument for the next piece of work.
What is being tested
The defining question of the archetype. Volunteering 'every proxy is gameable' and 'never per individual' unprompted is what separates a real practitioner from someone who read the DORA paper.
Reliability improved this quarter. How do you know it was you?
What you said
A strong answer
I would be sceptical of my own claim first, because reliability numbers improve for boring reasons at least as often as for good ones.
The things that move the number without anyone doing anything: traffic was lower, the release cadence slowed because of a code freeze or holidays, a noisy dependency happened to be stable, or a bad SLO stopped firing because we changed what we measure. Any of those produce a quarter that looks like success.
So the comparison I want is like-for-like. Same traffic mix, comparable deployment volume, same measurement definition. If the SLI definition changed at any point in the quarter, the comparison is void and I would say so rather than take credit.
Then I would anchor on a specific intervention rather than the aggregate. ‘We added a circuit breaker on the payments dependency in week three; incidents attributable to that dependency went from four in the prior quarter to zero, and it accounted for most of the improvement’ is a defensible claim. ‘Reliability improved twelve percent’ is not, because nobody can tell whether it was us.
And if I genuinely cannot separate the signal from the conditions, the right answer is to say that. An SRE who claims credit for a quiet quarter loses the ability to be believed the next time the number moves for a real reason, which matters far more than one quarter of looking good.
What is being tested
The control habit, in SRE vocabulary.
Your eval improved by two points. Convince me that is real.
What you said
A strong answer
Two points is inside the noise for most eval sets, so my first move is to establish whether it is a result at all rather than defend it.
Seeds and variance. How many runs? If it is one run per arm, I have no variance estimate and the honest answer is that I do not yet know. I would run several seeds and report the spread, not the best run — reporting the best is the most common quiet dishonesty in applied ML.
Set size. On a two-hundred-case eval, a two-point difference is roughly what you would expect from resampling alone. As a rule of thumb the margin near fifty percent is about one over the square root of n, so I would want to know n before saying anything.
Contamination. Did anything in the eval leak into training, directly or through a shared upstream source?
Ablation. Which single change produced it? If I remove that change, do the two points come back? A result I cannot attribute to a specific cause is a result I do not trust.
Does it hold out of distribution? A gain that appears only on the eval set and not on a held-out slice drawn later is usually overfitting to the benchmark.
If after all that it survives, I would still present it as ‘two points, with a spread of one to three across five seeds’ rather than as a headline — because the credibility of everything else I claim depends on not overselling this.
What is being tested
Empirical rigour, and whether you know your own result's failure modes.
How would you know this deployment succeeded, six months from now?
What you said
A strong answer
By having agreed the measure before we started, in writing, with the person who is paying — because a success measure defined at the end is negotiated rather than measured.
What I want is a business outcome with a baseline, not a usage number. ‘Average handling time on this ticket category falls from eleven minutes to under seven’ is a measure. ‘Eighty percent of agents use the tool weekly’ is not — people open things and abandon them, and adoption without effect is the most common way an AI deployment quietly fails while everyone reports green.
So concretely, in the first two weeks I would establish: what the metric is, what it is today, who owns it, and how we will read it in six months without a special project. If nobody can tell me today's number, that is the first deliverable, and I would rather find that out in week one than in month five.
Then I would name the failure mode explicitly with them: the system works, people use it, and the metric does not move — usually because the bottleneck was somewhere else in the process. Agreeing in advance what we would do in that case is uncomfortable and it is exactly what separates an engagement that lands from one that gets quietly wound down.
What is being tested
Whether you think in outcomes or in shipped features.
Offline AUC improved and the A/B was flat. Give me your hypotheses, in order.
What you said
A strong answer
I would give an ordered list, because the order is the answer — anyone can list five hypotheses, but knowing which to check first is the experience.
First, training-serving skew. By far the most common cause. A feature computed one way in the offline pipeline and another way at serving time — different windows, different null handling, a join that silently drops rows. I would check this by logging the actual feature vector at serving time for a sample of requests and diffing it against the offline computation for the same entity. If they differ, nothing else matters.
Second, the offline set is not representative. Evaluated on a random split when traffic is seasonal, or on historical data from before a product change.
Third, exposure bias. The training labels come from what the previous model chose to show. The new model may be better at ranking items the old one never surfaced, which offline data cannot reward and cannot even measure.
Fourth, metric mismatch. AUC is a global ranking measure; the business cares about the top three slots. A model can improve AUC by better ordering items nobody sees.
Fifth, insufficient power. The A/B may simply not have been able to detect the effect size — I would check what minimum detectable effect the sample supported before concluding it was flat.
In my experience it is skew about half the time, which is why I check it first and why I now diff feature vectors as a standing pre-launch step.
What is being tested
The signature question of the archetype. An ordered list with a stated first check beats a longer unordered one, and naming training-serving skew first is what a practitioner does.
Justify your team's headcount to a CFO.
What you said
A strong answer
I'd lead with the arithmetic and then immediately undercut it, because a CFO trusts a number that arrives with its own caveats far more than a clean one.
The arithmetic: if we have 300 engineers and we removed 20 minutes of waiting per engineer per day, that is 100 engineer-hours a day, which at a loaded cost is a number far larger than a six-person platform team. I would show the measurement behind the 20 minutes — build time before and after, with dates — rather than asserting it.
Then the honesty: that calculation assumes recovered time converts to output, and it does not convert one-for-one. So I'd give the range, name the assumption, and offer the harder evidence instead — change failure rate, time to restore, onboarding time to first production change. Those are outcomes rather than inputs, and they are much harder to argue with.
And I'd frame the alternative honestly, because that is what actually decides the conversation: without this team, this work does not disappear. It gets done 300 times, badly, by people who are not specialists, and it shows up as slower delivery and more incidents rather than as a line item. The question is not whether to pay for platform work; it is whether to pay for it once or three hundred times.
What is being tested
Whether you can argue in the CFO's language rather than the engineer's, and whether you volunteer the weakness in your own number.
Design a service handling 50k QPS at 99.9% availability.
What you said
A strong answer
I would do this with numbers out loud, because the non-abstract part is the exam — a component diagram with no arithmetic fails this round regardless of how good the architecture is.
Load. 50k QPS is the stated figure; I would ask whether that is peak or average, and if unstated assume it is average and apply a peak factor of two to three. Call it 150k QPS at peak, and say that I am assuming it.
Per-node capacity. This needs measuring, but for a simple stateless service I would assume a few thousand requests per second per node at around seventy percent CPU, and state that the number is an assumption that changes everything downstream. Say 3k, giving roughly 50 nodes to serve peak.
Failure domains. Three availability zones, each able to lose one node and still serve, and the fleet able to lose an entire zone. That means sizing each zone to carry the full load with a zone gone — so roughly 75 nodes rather than 50.
Headroom. Multiply by about 1.4 for growth and for the fact that autoscaling is not instant. Call it 105 nodes.
Then the binding constraint, which is the part that shows experience: it is rarely CPU. It is usually the datastore, connection limits, or a downstream dependency with its own rate limit. I would name which one I think it is and how I would verify.
And 99.9% is about 43 minutes of error budget a month, which is the number that decides whether this design is adequate — it means a single bad deploy can consume the quarter.
What is being tested
NALSD. Non-abstract is the exam — a component diagram with no arithmetic fails.
Estimate the GPU memory to serve a 70B model at batch 32 with 8k context.
What you said
A strong answer
I would work it out loud, stating assumptions, because the arithmetic is the exam rather than the final number.
Weights first. Seventy billion parameters at two bytes each in fp16 is about 140 gigabytes. That already exceeds any single accelerator, so we are sharded before we begin — and I would say that, because it frames everything after.
Then the KV cache, which is the term candidates forget. It scales with batch times sequence length times layers times hidden dimension times two, for keys and values, times bytes per element. For a model of this class at batch 32 and 8k context that lands in the tens of gigabytes — the same order as a meaningful fraction of the weights. The important property is that it grows linearly with both batch and context, so it is the term that decides your maximum concurrency, not the weights.
Then overhead — activations, fragmentation, the framework itself. I would add a working margin rather than pretend to precision.
So: multi-GPU, probably eight accelerators with tensor parallelism, and the cache is what limits how many concurrent requests we serve.
Then the part that shows operational experience: what I would do about it. Quantise the weights to int8 or fp8 and halve the largest term. Use paged attention so cache memory is not pre-allocated per sequence at maximum length. Cap context, because unbounded context is unbounded cost. And measure tokens per second per dollar rather than raw latency, since that is the number the business is actually buying.
What is being tested
Whether you reach for the KV cache unprompted. Most candidates give only the weights.
Design multi-tenant isolation for untrusted workloads.
What you said
A strong answer
The first thing I would establish is the threat model, because it decides everything and candidates who skip it design either far too little or far too much. Are these tenants merely separate customers who should not see each other's data, or are they running arbitrary code we do not trust?
If they run untrusted code, namespaces are not the boundary — they are an organisational construct sharing one kernel, and a kernel escape crosses them. The real options are separate node pools per tenant, a sandboxed runtime such as gVisor or Firecracker, or full VM isolation, and I would choose based on how much performance overhead we can accept.
Then the layers, none of which are optional. Network policy default-deny with explicit allows, so a compromised pod cannot reach a neighbour. Resource quotas and limit ranges, because noisy-neighbour resource exhaustion is a denial of service. Separate credentials and separate encryption keys per tenant, so one leaked key is bounded. Admission control to reject privileged pods, host mounts and unsigned images. And distinct storage, since a shared volume undoes all of it.
Then the part most candidates omit, which is how you know it works. I would write an isolation test that actively attempts cross-tenant access — a pod trying to reach another tenant's service, read its secret, mount its volume — and asserts refusal, running in CI. And I would periodically delete the guard and confirm the test goes red, because an isolation test that passes vacuously is worse than no test: it produces confidence with no basis.
What is being tested
Whether you know where the actual trust boundary is.
Your Terraform apply would have deleted a production database. How did your process allow that?
What you said
A strong answer
I would answer about the process rather than the person, because the interesting question is why the system permitted it.
The chain of failures that produces this. The plan was not read — or was read by someone who could not realistically parse two hundred lines of output to spot one destroy. Nothing distinguished destructive changes from additive ones in the review. State and blast radius were too coarse — one state file covering both the application and its data means a mistake anywhere can reach the database. It ran from a laptop rather than through a pipeline with an approval gate.
The fixes, in order of value. Put prevent_destroy lifecycle blocks on stateful resources, so the apply fails rather than proceeds — this alone stops the class. Split state so data infrastructure is separate from the compute that uses it. Run applies through CI with the plan posted for review and destructive changes highlighted, and require a second approver when the plan contains a destroy. Then the backstop: point-in-time recovery enabled and a restore that has actually been tested, because a backup nobody has restored is a hypothesis.
What I would not do is add a rule saying people should read plans more carefully. That is not a control — it is a hope, and it will fail again on a Friday afternoon.
What is being tested
Whether you think in blast radius. Blaming the person fails this.
Your agent's p50 is fine but users say it is slow. Where do you look?
What you said
A strong answer
Instrument the whole turn, not each call. If a turn chains four calls that each have a 1% tail, roughly 3.9% of turns hit at least one tail — so users experience worst-case latency about four times more often than any per-call dashboard suggests. Then look for sequential calls that could run in parallel, and timeouts set above the downstream p99.
What is being tested
The one genuinely counterintuitive result in agent engineering.
Cut inference cost 40% without hurting quality. What do you try, in what order?
What you said
A strong answer
I would sequence by cost-to-try, and gate every step on the eval, because the failure mode is making it cheaper and quietly worse.
Caching first, because it is nearly free. Exact-match caching for repeated queries, and prompt-prefix caching where the provider supports it — in retrieval systems a large share of the prompt is identical across calls.
Then trim the context. Most systems resend far more than they need: full conversation history, over-retrieved chunks, verbose system prompts. This is usually the single largest win and costs nothing in quality when done carefully.
Then routing. Send easy requests to a small model and reserve the large one for the hard tail, using a cheap classifier or a confidence signal. Typically most traffic is easy, so this is where the step change is.
Then batching, if we self-host — continuous batching substantially improves throughput per GPU.
Then quantisation, int8 or fp8, measuring quality at each step.
Distillation last, because it is a project rather than a change.
Throughout I would measure cost per resolved task, not per call, since retries and failures are real cost and a cheaper model that needs three attempts is not cheaper. And I would stop as soon as the eval moves beyond the noise floor, then report both numbers — what we saved and what it cost in quality — because a cost reduction presented without its quality impact is not a result anyone should accept.
What is being tested
Whether you sequence by cost-to-try, and whether you gate each step on an eval.
Your observability bill is larger than your compute bill. What do you cut first?
What you said
A strong answer
The cause is almost always cardinality, so I would go there first rather than proposing to retain less.
Every unique combination of label values on a metric creates a separate time series. Adding one high-cardinality label — user ID, request ID, full URL path, container ID — multiplies the stored series by the number of distinct values, so a single well-meaning line of code can turn a hundred series into ten million. I would find the top metrics by series count, and I would expect a handful to account for most of the bill.
The fix is to move that data to the right signal rather than delete it. High-cardinality context belongs in traces and structured logs, which are sampled and are designed for it, not in metrics, which are designed to be aggregated. So: strip high-cardinality labels from metrics, keep them as trace attributes, and use tail-based sampling so we keep the slow and failed traces rather than a random one percent — head sampling throws away exactly the traces you need during an incident.
Then the operational hygiene: a per-team budget with visibility, so cost lands where the decision is made; a review on new metric definitions, the same as a schema change; and retention tiers, because ninety days of full-fidelity data is rarely justified when thirty days plus aggregated history serves the same purpose.
And I would state which queries we lose, explicitly. A cut presented as free is one nobody believes, and being honest about the trade is what gets it approved.
What is being tested
Whether you understand the mechanism rather than just proposing to retain less.
How would you price a feature whose cost varies per use?
What you said
A strong answer
I would start from cost per resolved task rather than per API call, because retries, failed turns and human escalation are real cost and pricing off the call makes the unit economics look better than they are.
Then I would model the distribution rather than the mean, which is the mistake that kills AI feature margins. A small tail of very expensive sessions — long conversations, large documents, users who retry — can dominate total cost while the average looks fine. So I want the p50, p95 and p99 cost per user per month, and I would price against the p95.
Then choose a structure that bounds the downside. Usage tiers with an included allowance and clear overage is the most honest and the easiest to explain. Seat pricing with a fair-use cap works when usage correlates with headcount. Pure per-token pass-through is transparent but makes budgeting impossible for the customer, which loses enterprise deals.
Two things I would insist on regardless. Instrument cost per customer from day one, because you cannot retrofit that and without it you will not discover the loss-making segment until it is large. Build a circuit breaker — a per-account spend cap that degrades rather than bills unbounded — since the first real incident in this space is usually a runaway loop on one account.
And I would say plainly that inference prices fall over time, so a margin that is thin today may be fine in a year — but that is a bet, and it should be a stated one rather than an assumption.
What is being tested
Whether you know the denominator is the whole question.
Walk me through an incident you were part of.
What you said
A strong answer
I would tell it in a fixed shape, because incident stories sprawl and the interviewer is listening for decisions rather than diagnosis.
Impact first, in user terms: checkout failed for roughly eight percent of users for thirty-one minutes. Not ‘a pod was OOMKilled’ — leading with the mechanism signals that I think the system is the point.
How we found out, honestly. If a customer told us before monitoring did, I say so, because that is the most common real answer and it is the most useful finding.
The decision under uncertainty, which is the centre of the story. What I did before I understood the cause and why. Usually that is rolling back without knowing the reason, which is almost always correct and which candidates apologise for — I would not. Mitigate first, understand second.
Root cause in two sentences. This is the part I want to spend ten minutes on and should spend one.
What changed, and whether it shipped. The strongest ending is an action item that landed, with a date. The second strongest is honest: it is still in the backlog, here is why, and here is what we did instead.
If I have never been on call, I would use what I have — a build I broke for the team, a migration that corrupted staging data — and say plainly what it is. The structure is identical and borrowed war stories fall apart under two follow-up questions.
What is being tested
The decisions, not the diagnosis. Spending eight minutes on root cause is the common error.
How would you run a blameless postmortem with the person who caused the outage in the room?
What you said
A strong answer
The question is not about my feelings on blame; it is about whether I can hold two things at once — the outcome was bad, and this person is not the cause.
What I would actually do in the room is move the frame from the person to the system, out loud, in the first few minutes. Not ‘why did Priya run that command’ but ‘why could a single command drop a production table with no confirmation, no dry run, and no verified backup?’ That reframing is not politeness. It is the only version of the question with an actionable answer, because we cannot fix Priya and we can fix the command.
The specific technique I use is to ask what information they had at the time, not what we know now. Almost always the decision was reasonable given what was visible — the runbook said this, the dashboard showed that, the alert did not fire. The gap between what they could see and what was true is the finding, and it is a much richer one than ‘be more careful’.
I would also protect them procedurally: they walk through the timeline, not the incident commander narrating over them, and nobody outside the room gets a version that names them.
And I would not let blameless become accountability-free. The review still ends with named owners and dates for the actions. The distinction is that the action is on the system, and the owner is whoever owns that system — which is frequently not the person who typed the command.
What is being tested
Whether you can hold ‘bad outcome’ and ‘not this person's fault’ at once.
Your error budget is exhausted in week two of the quarter. Product wants to ship. What happens?
What you said
A strong answer
The policy should already exist, and it should have been agreed when nobody was under pressure — that is the entire point of writing it down in advance. So my first answer is: we do what the policy says, and I would read it out.
A workable policy looks like this. Budget exhausted means a feature freeze for that service — reliability work only — until the trailing window recovers. There is a named exception path, usually a specific executive who can override, and the override is recorded. And there is a clear statement of what does not count against the budget, because arguments about attribution consume more time than the outages do.
If no policy exists, then I would say that plainly, and the deliverable becomes writing one rather than winning this particular argument. Trying to enforce an unwritten rule in the middle of a release cycle is how SRE teams get a reputation for being obstructive.
What I would actually do in the room: separate the two questions. Is the budget real — are we genuinely burning user-visible reliability, or is the SLO badly chosen and firing on something users do not notice? That happens often and the honest answer may be to fix the SLO. And if it is real, what is the smallest thing that gets us back under — because the goal is not to block the release, it is to stop the burn, and sometimes a rollback of one change achieves that.
I would also be honest that this is a negotiation with a power imbalance, and that the SRE who wins it every time is not doing the job well either.
What is being tested
The hardest SRE round. There is no clean answer; they want your reasoning and escalation path.
Your agent calls a tool that fails silently and returns an empty result. How do you detect it?
What you said
A strong answer
Not with a better prompt — that is the answer I would rule out first, because it is where most candidates go. A model cannot reliably detect that a tool lied to it, and instructing it to ‘check whether the result looks empty’ is asking a probabilistic system to do a deterministic job.
The fix is at the boundary. Validate every tool response against a schema and treat an empty or malformed result as an error, not as a value. A search that returns zero results is a legitimate outcome; a search that returns null because the API timed out is not, and the two must be distinguishable in the response type. Trace every tool call with arguments, status, latency and result size, so that when output quality drops you can see that the retrieval tool has been returning empty for six hours. Add an eval case that exercises exactly this — a stubbed tool that fails silently — so the behaviour is a regression test rather than an incident.
Then the design decision, which is the part interviewers actually want: what should the agent do? Retry with backoff, fall back to a degraded path, or surface the failure to the user. Silently continuing is the worst option and it is the default, because a model handed nothing will confidently produce something. My preference is to fail loudly to the user for anything consequential, and degrade gracefully with an explicit caveat for anything informational.
What is being tested
Whether you reach for observability and typed boundaries rather than prompt engineering. It also tests whether you know that a model given no context will confidently invent one.
The customer's data is far worse than they told us. Walk me through that conversation.
What you said
A strong answer
I would lead with what it means for them, not with what is wrong with their data, because the second framing makes it sound like their fault and closes the conversation.
So: ‘Based on what we have found, the March date is no longer realistic. Here is what I think is achievable by March, here is what moves to May, and here is what I need from you to hold that.’ Then the specifics — not ‘the data is bad’ but ‘forty percent of records are missing the field we key on, and the date format changes partway through 2023.’ Concrete beats damning.
Three things I would insist on. Bring the replan to the same meeting. Bad news without a plan is an emergency; bad news with options is a decision. Say what is still on track, because the natural reaction is to assume everything is now at risk. Be explicit about what I need — usually access to someone who knows why the data looks like that, which is almost always a person rather than a document.
And I would deliver it early. The cost of telling a customer in week two that the timeline slipped is a difficult meeting. The cost of telling them in week eight is the relationship, because by then they have told their own management the original date. Every engagement I have seen go badly went badly because someone hoped it would resolve itself.
What is being tested
Whether you can deliver bad news without sugar-coating it and without assigning blame — and whether you bring the replan rather than just the problem.
What would you deprecate in your first ninety days, and how would you know you were wrong?
What you said
A strong answer
I would name something specific and unglamorous, with a date and a migration path, because a vague answer here reveals someone who has never actually turned anything off.
The candidate I would pick is the thing with the worst ratio of maintenance cost to users — typically an internal tool with three consumers that one person understands, or a second way of doing something the platform already does once. Two ways to deploy is worse than either way alone, because every subsequent platform change has to be built twice.
The plan: publish the date once, well ahead. Do the migration work myself rather than delegating it — write the codemod, open the pull requests, and ask consumers only to review. Provide the replacement first and make it better, not merely equivalent. Then hold the date, because a deprecation date that slips teaches everyone that the next one will too, and the migration after this one becomes much harder.
Then the half most candidates skip — how I would know I was wrong. The signals: a team blocked with no viable alternative, a use case I did not know existed surfacing during migration, or a cost landing on someone else's budget that I had not accounted for. If any of those appear, I stop and re-scope rather than pushing through, and I say so publicly. A platform team that never reverses a deprecation is not being careful, it is not listening.
What is being tested
Whether you can kill something you did not build, and admit being wrong about it.
What is the strongest argument against your own last result?
What you said
A strong answer
I would give a real one, because claiming there isn't one is the weakest available answer and interviewers hear it constantly.
The strongest arguments against most results fall into a few families, and I would name whichever genuinely applies to mine. The baseline was not tuned as hard as the method — I spent three weeks on my approach and an afternoon on the comparison, which is the most common and least discussed flaw in applied work. Contamination — some evaluation data may have been visible during training, and depending on the source I may not be able to fully rule that out. The effect is scale-specific — it holds at the size I tested and there is no evidence it survives an order of magnitude either way. The metric rewards my method's failure mode — if my approach produces longer outputs and the metric correlates with length, some of the gain is an artefact.
Then what I did about it. For the baseline concern, the honest answer is usually to go back and tune it properly, and I would say whether I did. For contamination, decontamination checks and a held-out set drawn from after the training cutoff.
And I would say what would change my mind — the specific experiment that would falsify the result. A researcher who cannot name that has not stress-tested their own work.
What is being tested
Intellectual honesty, and whether you have actually stress-tested your own work.
Tell me about a deal you lost.
What you said
A strong answer
I would give a specific technical reason, because ‘we lost on price’ is the answer everyone gives and it tells the interviewer nothing.
The shape that lands: what they were running, what we proposed, and the concrete thing that made ours the wrong answer. For example — they had a large existing Kafka footprint and three years of operational knowledge around it; our platform would have replaced that, and once we modelled the migration effort honestly the three-year saving no longer covered it. I said so in month two rather than month six, and we lost the deal but kept the relationship, and they came back eighteen months later for a different workload.
Then what I changed afterwards, because that is the half candidates skip. In that case I built a migration-cost model into our discovery so we could have that conversation in week one rather than after we had consumed a lot of the customer's time. It meant qualifying out earlier, which felt like losing pipeline and was actually the opposite.
Two things I would avoid. Blaming the account executive or the product, because it reads as someone who does not own outcomes. And picking a loss where we were obviously going to lose — the useful story is one that was genuinely winnable, because that is where the learning is.
What is being tested
Whether you have real deal experience and can be honest about losing.
Explain prompt injection to an executive who thinks it is just input validation.
What you said
A strong answer
Start with what makes them different rather than what they have in common. SQL injection is solved because a parameterised query gives the database two separate channels — it is told which bytes are code and which are data, so a malicious value can never become an instruction. A language model has one channel. Instructions and data arrive as a single stream of tokens with no marking to distinguish them, so text inside a document the model reads is indistinguishable from text you sent. There is no parameterised query available, and no filter closes it, because the filter would have to separate ‘text describing an instruction’ from ‘an instruction’ — the same undecidable problem one level up. Then land the consequence: because we cannot prevent it reliably, the control moves from the input to the blast radius. Scope tools to least privilege, require human approval on irreversible actions, keep the component that reads untrusted content separate from the one holding write credentials, restrict egress, and log every tool call. Those are conventional controls we already know how to specify, which is the reassuring part.
What is being tested
Whether you understand the mechanism rather than the vocabulary — and whether you can brief without either minimising or catastrophising.
You have been asked to audit an AI system. Where do you start, and what evidence do you request?
What you said
A strong answer
Start with the inventory entry: does the system appear in a register, with a named owner, a stated purpose, and a tier based on the consequence of it being wrong? Then request the evidence set. A written statement of intended and prohibited use. The evaluation suite — the golden set with a date, the scoring method, the recorded baseline and the last several regression runs. A change log tying model and prompt versions to eval runs and approvals. A data-flow diagram covering the model provider and the logs. The tool permission list. Tool-call logs carrying identity, arguments, result and timestamp. Approval configuration plus a sample of real approvals. Monitoring thresholds and any incidents raised. If I could ask only one thing, it would be this: take one real output from last month and reconstruct it — the exact input, the retrieved context, the model and prompt versions, the tool calls, the output and who saw it. That single request exercises logging, versioning, retention and traceability at once. And ‘it’s a black box’ is a finding, not an answer: the model’s internals are opaque but the system around it is ordinary software with ordinary logs.
What is being tested
Whether you can audit AI without understanding neural networks — which you can, because the evidence is systems evidence.
The business wants to use a public AI chatbot. Do you allow it?
What you said
A strong answer
Yes, with conditions — and I would open by saying that a ban is the higher-risk answer, because that reframes the whole conversation from permission to design.
Prohibition does not produce abstention. It produces unmonitored use on personal devices, outside our logging, DLP, contracts and incident response. The organisation takes all of the exposure and gets none of the controls, and we find out during the incident. Samsung is the reference case: the ban followed the leak and pushed usage off the corporate estate rather than stopping it.
The conditions I would attach. An enterprise agreement that contractually excludes training on our inputs, with retention specified and sub-processors listed. SSO, so usage is attributable to a person rather than anonymous. A stated data classification ceiling with concrete examples rather than abstractions — people cannot apply ‘no confidential data’ but they can apply ‘no customer records, no source code, no unreleased financials’. Logging and retention decided deliberately, since the prompt stream is now a data store. And a one-page acceptable-use policy people will actually read.
Then I would publish it as the paved road and commit to answering exception requests in days. A two-week turnaround is functionally a prohibition for a team on a sprint, and speed is a security control because it determines whether people come to us at all.
And I would say out loud which risks I am not worried about, because clearing five of seven concerns is what earns credibility on the two that matter.
What is being tested
Whether you are the function that enables or the function that gets routed around.
A team says they want to train a model on customer data. What do you ask?
What you said
A strong answer
First, which of four things they mean, because they differ by six orders of magnitude in cost and, more importantly, in whether the data can ever be removed. Prompting sends data in a request — deletion is a log-retention question. RAG puts documents in a search index we control — deletion works, and the index inherits the source classification. Fine-tuning absorbs the data into weights, and it cannot be selectively un-learned; the honest answer to an erasure request is retrain without it. Pre-training is not a decision they are making. So my questions are: are we changing the weights or changing the request; if a customer asks for erasure next year what exactly do we do; where does the data sit in the meantime; and whose data shares that location. Usually the answer is that they want current facts rather than changed behaviour, which means retrieval is both the governance-friendly option and the technically better one — fine-tuning teaches style and format, and is an expensive, stale way to teach facts.
What is being tested
Whether you can prevent an expensive mistake by asking one clarifying question.
Walk me through what happens when an employee changes department.
What you said
A strong answer
In theory, the old entitlements are removed and the new ones granted. In practice only the second half happens, and that asymmetry is the whole problem: grants are automated and welcomed, revocations are a ticket somebody has to remember to file. After two moves a person holds the union of three roles, which is privilege creep, and it is how a phished account in support reaches finance systems. What I would check: are grants derived from a role, or cloned from a colleague — ‘give them what Priya has’ propagates one person’s accumulated access across the org. Does a role change fire a revocation workflow at all, or only an addition. What is the measured time from HR change to last stale entitlement removed. And I would look hardest at contractors and service accounts, because neither has an HR record, so neither has a lifecycle, and both accumulate silently outside the metrics anyone reports.
What is being tested
Whether you know that mover is the weak stage. Most candidates talk about joiner and leaver.
Access reviews keep getting rubber-stamped. Fix it.
What you said
A strong answer
The behaviour is rational, so I would change the task rather than exhort the reviewers. A manager facing 400 entitlements with cryptic technical names and a deadline will approve all of them, and a rubber-stamped review is worse than none because it produces evidence of a control that is not operating. Four changes. Review by exception — show what changed and what is unused, not the full list. Lead with usage data, because ‘these 40 entitlements have not been used in 180 days’ is a decision a human can actually make, whereas ‘is this appropriate?’ is not. Make revocation the default for unused access with a one-click path to reclaim it, which inverts the burden. And wherever possible replace review with time-bounded grants, so access expires without anyone deciding — converting an unreliable human control into an automatic one. I would measure revocation rate, not completion rate; a review that revokes nothing has told you nothing.
What is being tested
Whether you fix the system or blame the users. Also whether you know completion rate is a vanity metric.
What is the difference between CyberArk and HashiCorp Vault?
What you said
A strong answer
They overlap and both have grown into the other’s territory, but the centre of gravity differs and the distinction holds in practice. CyberArk’s classic problem is the human privileged administrator — vaulting shared admin credentials, brokering and recording sessions so the human never handles the secret, rotating after use, and producing the compliance evidence auditors ask for. It is usually bought by the security or IAM team. Vault’s classic problem is the application that needs a database password at runtime — it is usually bought by platform engineering, and its most distinctive capability is dynamic secrets: it creates a credential on demand with a short lease, so the credential your service is using did not exist an hour ago and will not exist in an hour. If I had to compress it: CyberArk is about humans reaching systems, Vault is about workloads reaching systems. And I would add that the real endpoint for both is workload identity, where the platform attests what the workload is and issues a short-lived token — the best secret is the one that does not exist.
What is being tested
Whether you have actually worked with them or only read the marketing. The dynamic-secrets point is the tell.
We deployed PAM two years ago. How would you tell whether it is working?
What you said
A strong answer
I would not ask whether it is deployed, because it is — the question is what fraction of privileged activity actually goes through it, and in my experience nobody has computed that number.
PAM programmes almost never fail on technology. They fail because the vault goes in, the administrators keep their standing accounts ‘for emergencies’, and eighteen months later a large share of privileged work still bypasses the system entirely. The tool is operating; the control is not.
So the metrics I would want. Coverage: what percentage of privileged sessions last month were brokered, and what is the trend — this is the number that matters and it is usually uncomfortable. Standing privilege count: how many permanent privileged accounts still exist, trending down, because the goal was never a better password store, it was the removal of the standing credential. Exposure window: for accounts on just-in-time, the median elevation duration — if people are elevating for eight hours, they have recreated standing access with extra steps. Break-glass usage: how often, by whom, and did each use trigger a review.
Then I would look for the bypass paths, because they always exist: local accounts on appliances, service accounts with interactive login enabled, cloud console access outside the broker, and the emergency account everyone knows the password to.
If nobody has the coverage number, that absence is itself the finding, and I would make producing it monthly the first deliverable.
What is being tested
Whether you measure control operation rather than control existence.
When would you choose ABAC over RBAC?
What you said
A strong answer
I would frame it by how each fails rather than which is better, because real systems are hybrids. RBAC fails by role explosion — every exception becomes a new role instead of a policy, and once you are past a few hundred roles for a few hundred people the model no longer describes anything a human can reason about. ABAC fails by becoming unanswerable: nobody can say who can reach a system without running the engine, which is uncomfortable for audit. Hybrids fail worst, with two sources of truth where a role grants and a policy denies and nobody knows which wins. I would reach for attributes when the decision genuinely depends on request context — data sensitivity, device posture, location, time — because encoding that in roles is what causes the explosion. And I would push toward externalised policy-as-code with OPA, Cedar or OpenFGA, because policy in version control with tests and commit history is far better evidence than permissions hand-configured across twelve consoles. The audit question then is not ‘do you have a policy engine’ but ‘which code paths bypass it’.
What is being tested
Whether you think in failure modes. Naming role explosion unprompted is the strong signal.
How should an AI agent authenticate to internal systems?
What you said
A strong answer
Not with a shared service account, which is the default and the wrong answer. An agent on a shared account is unattributable — you cannot tell which agent, which user’s request, or which run took an action, so you cannot investigate. The agent needs its own identity, and it should act on behalf of the requesting user with authority scoped per request rather than holding standing privilege. That is the confused deputy problem: if the agent holds a broad credential and any user, or any document it reads, can influence what it does, then that party has effectively borrowed the credential. The established pattern is delegation — OAuth token exchange, on-behalf-of flows — issuing short-lived, task-bound credentials. Then every tool call must be attributable to an agent identity, a run, and an initiating human. This is becoming a specialism in its own right; the identity vendors are already hiring for it explicitly.
What is being tested
Whether you connect classic identity theory to the new problem. The confused-deputy framing is what senior interviewers listen for.
How do you get developers to fix the vulnerabilities you find?
What you said
A strong answer
My first assumption is that this is my failure rather than theirs, because four thousand findings means zero fixes and the team has correctly learned that the queue is noise.
Establish the signal-to-noise ratio before asking anyone for anything. Take fifty findings at random and work out how many are genuinely exploitable in our context — reachable code path, real data, actual privilege. If that number is under ten percent, the scanner configuration is the problem and no amount of accountability will fix it. Tune, suppress with a documented reason, and be prepared for the count to drop by ninety percent. That looks like backwards progress and it is the necessary first move.
Change where the finding appears. A vulnerability in a quarterly PDF is not actionable. The same finding as a pull-request comment on the offending line, with a suggested diff, gets fixed before merge because it is cheaper to fix than to argue with.
Fix classes, not instances. If the same injection pattern appears in forty places, forty tickets is the wrong answer. A safe-by-default library plus a lint rule that blocks the pattern removes the class and prevents the next forty.
Measure time-to-fix by severity, not findings closed, because findings-closed rewards me for generating noise and tells me nothing about risk.
And I would go and sit with the team that has the most open criticals rather than escalating. Escalation works once; understanding why they cannot fix it — usually an unowned service or a dependency they cannot upgrade — works repeatedly.
What is being tested
Whether you have shipped an AppSec programme or only run scans. The findings-count point is the tell.
Your SIEM produces 500 alerts a day and the team is drowning. What do you do?
What you said
A strong answer
I would treat it as a precision problem and start by measuring, because the team's behaviour is already telling me the answer.
The number I want first: for a sample of alerts, what fraction led to any action at all? If it is under a few percent, the pipeline is producing noise, the analysts have rationally learned to ignore it, and a real detection will be missed inside that noise. That is the actual risk, not the volume.
Then tune by volume. Alert distribution is always heavily skewed — a handful of rules generate most of the load. I would take the top ten by volume and for each decide: tune it, convert it to a correlation input rather than an alert, or retire it. Retiring is legitimate and under-used; most SIEMs accumulate rules and never remove any, so the pipeline reflects every threat anyone worried about since 2019.
Then detection-as-code. Rules in version control, with code review, and a test that includes a known-true positive which must fire. Without that, tuning silently breaks detections and nobody finds out.
Then a runbook per alert. An alert nobody knows how to action is a notification, not a detection. If I cannot write the runbook, the rule should not exist.
I would report alert-to-action ratio and median time to triage, and specifically not alert volume, because volume rewards exactly the behaviour that created this situation.
What is being tested
Whether you optimise for signal or for coverage theatre.
How would you secure a multi-tenant SaaS platform on Kubernetes?
What you said
A strong answer
I would fix the threat model first, because it determines whether this is a two-week job or a re-architecture. Are tenants merely customers who must not see each other's data, or are they running code we do not trust?
If untrusted code is in scope, namespaces are not a security boundary — they are an organisational one over a shared kernel, and a container escape crosses them. The real options are a node pool per tenant, a sandboxed runtime such as gVisor or Firecracker, or full VM isolation, chosen on how much overhead we can accept. I would say which I would pick and why.
Then the layers, none optional. Network policy default-deny with explicit allows. Resource quotas and limit ranges, since noisy-neighbour exhaustion is a denial of service. Per-tenant credentials and per-tenant encryption keys, so one leaked key is bounded. Admission control rejecting privileged pods, host mounts and unsigned images. Separate storage, because a shared volume undoes everything above it.
Then the part candidates omit: how you know it works. I would write an isolation test that actively attempts cross-tenant access — reaching another tenant's service, reading its secret, mounting its volume — and asserts refusal, running on every change.
And I would periodically delete the guard and confirm the test goes red. An isolation test that passes vacuously is worse than no test, because it manufactures confidence with no basis, and UUID-keyed data will make many naive tests pass whether or not the control exists.
What is being tested
Whether you know namespaces are not a security boundary, and whether you verify isolation rather than assume it.
How do you quantify a risk nobody has data for?
What you said
A strong answer
I would structure the uncertainty rather than invent a number, and I would be explicit that the output is a range with stated assumptions rather than a figure.
Decompose into things people can actually estimate. Nobody can estimate ‘the risk of a breach’. People can estimate how often the triggering event occurs, how likely it is to succeed given our controls, and what the impact range looks like if it does. Decomposition also surfaces where the disagreement really sits, which is usually the most valuable output of the whole exercise.
Elicit ranges, not points. Experts are poorly calibrated on single numbers and much better at ‘between X and Y, and I would be surprised if it were outside that’. I would ask several people independently before letting them discuss, because the first person to speak otherwise anchors the room.
Run the ranges rather than the midpoints. A simple Monte Carlo over the distributions gives you a loss curve, and the tail is the part the board actually cares about. Multiplying midpoints hides exactly the scenario that matters.
State what would change the estimate. A risk assessment that cannot be updated is a document, not a control. I would name the two or three pieces of evidence worth collecting and what each would do to the range.
And I would resist the pressure to produce a single red-amber-green rating, or at least insist that the rating carries the range behind it — because the moment it becomes a colour, the reasoning is discarded and only the colour is remembered.
What is being tested
Whether you can be rigorous without pretending to precision you do not have.
Management disagrees with your finding. What do you do?
What you said
A strong answer
I would separate three things that get conflated, because they need different responses and arguing across them is how these conversations go badly.
Do we disagree on the facts? That is settleable. We go back to the evidence together, and if I am wrong I correct the finding quickly and visibly — that is precisely what buys me credibility on the findings I do not withdraw. If the evidence stands, I restate it plainly and stop arguing, because repetition does not add weight.
Do we disagree on the significance? This is a legitimate disagreement about risk appetite, and management may well be right — they see business context I do not. My job is not to win it. My job is to make sure the decision is informed and recorded.
Do we disagree on the remediation? Usually the most negotiable. Timing, sequencing and approach are all open provided the control objective is met, and being flexible here is how I earn firmness elsewhere.
What I will not do is soften a factual finding to avoid friction, because a finding that bends under pressure is worthless to everyone including management — they need to be able to trust that what I report is what I found.
And in every case the residual risk gets accepted explicitly, in writing, by someone with the authority to accept it. That is not bureaucratic self-protection. An unrecorded disagreement is the worst outcome for both sides, because in eighteen months nobody remembers that a decision was made deliberately.
What is being tested
Whether you can hold a position without being rigid, and whether you know acceptance must be documented.
Our CI takes 45 minutes. Fix it.
What you said
A strong answer
I'd start by measuring where the 45 minutes actually goes, because the answer is almost never where people assume. Instrument the pipeline per stage and look at the distribution rather than the mean — a p50 of 45 minutes and a p95 of 45 minutes are different problems. Then I'd work in the order of cheapest-to-try. First, caching. Most pipelines rebuild dependencies from scratch on every run; a warm dependency cache and a build cache keyed on content hash often removes half the time for a day's work. Second, parallelism. Find what is sequential but need not be — lint, typecheck and unit tests almost never depend on each other. Third, test selection. Run the tests affected by the diff on every push and the full suite on merge to main; this is where the biggest wins are, and it needs a dependency graph, which is why Bazel and Nx exist. Fourth, the machine. Bigger runners are the least elegant fix and sometimes the most cost-effective one — an engineer waiting 45 minutes costs more than a larger instance.
Two things I'd insist on. I'd set a budget and alert on it, because pipelines regress silently as people add steps, and without a budget you will be back here in six months. And I'd measure the thing that actually matters to developers, which is not pipeline duration but time from push to merged — that includes queueing, flaky reruns and waiting for a reviewer, and it is frequently dominated by the parts that are not the build.
What is being tested
Whether you measure before optimising, and whether you order by cost-to-try. Jumping straight to 'we should use Bazel' is the common failure — it may be right eventually and it is never the first move.
You built a tool. Nobody adopted it. Diagnose.
What you said
A strong answer
I'd treat it as a product failure and go and watch people work, because the honest answer is almost always that the tool solved a problem I had rather than one they had. The diagnosis has a short list. Was the pain real? I may have fixed something that annoyed me and cost them ten minutes a month. Was the switching cost higher than the saving? A tool that saves five minutes but takes an hour to adopt is correctly rejected by a rational engineer with a deadline. Did they know? Internal tools have no marketing, and the usual outcome of an announcement in one channel is that nobody sees it. Was it worse on their real case? Internal tools are typically tested on the clean path and encountered on the messy one. Did I make them change something they cannot change? If adoption requires a team to alter their release process, that is not a tool decision, it is a political one.
Then I'd pick the three loudest teams, sit with one of them for a morning, and either fix the specific blocker or kill the tool. Killing it is a legitimate outcome and saying so in an interview is a strength — platform teams that cannot kill anything accumulate a museum of half-adopted tools, and the maintenance of that museum is what stops them building the next useful thing.
The durable fix is to stop shipping tools nobody asked for: find the team with the pain, build it with them, and let them be the reference customer for the next five.
What is being tested
Whether you take responsibility for adoption. Blaming developers for not adopting is the fastest way to fail this round, and it is very common.
Design an internal developer platform for 300 engineers.
What you said
A strong answer
I'd start by refusing to design it in the abstract, because the failure mode of this question is a beautiful architecture nobody uses. The first thing I'd do is find the three things engineers actually wait for — typically getting an environment, getting a database, and getting to production — and design for those, not for a reference architecture.
The shape I'd aim at is a paved road, not a walled garden. One well-supported way to create a service that gets you a repository, a pipeline, a deployment, monitoring, alerting and an on-call rotation on day one. Crucially it must be possible to step off the road: teams with genuine reasons to differ should be able to, and the platform's job is to make the road so obviously better that almost nobody wants to.
Concretely: a service template that scaffolds the whole thing; a self-service way to get infrastructure with sensible defaults and guardrails rather than a ticket queue; golden paths documented as runnable examples rather than wiki pages; and a catalogue so people can find what exists. I'd treat the platform as a product with a roadmap, internal customers and a support rotation.
What I would not do first: build a portal. Portals are the most visible part and the least valuable; teams adopt a platform because it removes waiting, not because it has a nice front page. And I'd measure adoption from day one — percentage of services on the paved path, and time from repository creation to first production deploy — because those two numbers tell you whether any of this worked.
What is being tested
Whether you design for real bottlenecks or from a reference architecture, and whether you know 'escape hatch' is a feature rather than a compromise.
What is your view on DORA metrics?
What you said
A strong answer
Useful, widely misused, and the misuse is predictable. The four — deployment frequency, lead time for change, change failure rate and time to restore — are good because they are outcomes rather than activity, and because they come in a pair that resists gaming: the two speed metrics are held honest by the two stability metrics, so you cannot buy one with the other.
Where it goes wrong. Using them to compare teams. A payments service and an internal dashboard should not have the same deployment frequency, and ranking them produces theatre. Treating the elite band as a target. It is a description of what some organisations achieve, not a goal every team should pursue; daily deploys for a system with a quarterly release window is a solution to a problem that does not exist. Measuring them without acting. The metrics are a diagnostic, and a dashboard nobody uses to choose work is decoration.
What I actually use them for: as a trend within a single team, to find the constraint. If lead time is long but deployment frequency is high, the queue is upstream in review or approval. If change failure rate is climbing while speed is flat, something has degraded in test coverage. The number is the pointer, not the goal — and I'd pair all four with a developer survey, because they measure the delivery pipeline and say nothing about whether the local environment takes a day to set up.
What is being tested
Whether you have used them or only read about them. Naming the comparison-between-teams trap unprompted is the tell.
Your test suite is flaky. Engineers have started re-running until green. What do you do?
What you said
A strong answer
The re-running is the emergency, not the flakiness, because it means the suite has stopped being a signal — a genuine failure now looks exactly like a flake, and something real will ship. I'd say that first.
Then: measure and rank. Record every test result with its outcome and identify tests that both pass and fail on the same commit. Flakiness is always concentrated — a small number of tests generate most of it. Quarantine aggressively. Move the top offenders out of the blocking suite immediately, into a non-blocking job, with an owner and a date. This is uncomfortable and it is correct: a quarantined test is honest about providing no signal, whereas a flaky blocking test actively destroys signal. Fix by cause, not by test. Flakes cluster into a few causes — time and timezone dependence, shared state between tests, real network calls, and async waits implemented as sleeps. Fixing the cause fixes twenty tests. Then make it not come back: a flake budget, a dashboard, and a rule that a newly flaky test is quarantined automatically rather than tolerated.
I'd track one number publicly: the percentage of pipeline runs that failed for a reason unrelated to the change. Getting that under about one percent is what restores trust, and until it is, no amount of asking people to stop re-running will work — they are behaving rationally.
What is being tested
Whether you understand that re-running is a rational response to a broken signal, and whether you will quarantine. Candidates who insist on fixing every flake before quarantining leave the suite untrusted for months.
How do you decide whether to build an internal tool or buy one?
What you said
A strong answer
I'd start from the fact that build is systematically underestimated, because people cost the first version and not the next five years of maintenance, on-call, upgrades and the person who leaves. So my default is buy, and build needs to clear a bar.
The bar has three parts. Is this differentiating? If the capability is part of how we win, build. If it is table stakes that every company needs, buy — nobody has ever won a market on the strength of their internal CI. Is the fit genuinely poor? Teams frequently claim their requirements are unique when they mean unusual and negotiable. I'd want to see two vendors actually evaluated against real workflows, not a feature grid. Can we carry it? Ownership means a named team, on-call, and a maintenance budget in perpetuity. If nobody will commit to that, the honest answer is buy, because an unmaintained internal tool is worse than a vendor.
Then two nuances. Buy does not mean no engineering — integration and glue are real work and get underestimated on the buy side. And the most common good answer is neither: assemble open-source components with a thin internal layer, which gets you fit without owning a product.
I'd also make the decision reversible where possible — keep the data model ours, keep the abstraction thin — so that being wrong costs a migration rather than a rewrite.
What is being tested
Whether you cost the maintenance, and whether you spot that 'our requirements are unique' is usually false.
Monorepo or polyrepo?
What you said
A strong answer
It depends on which pain you'd rather have, and I'd say so rather than picking a side, because both work at scale — Google and Meta run monorepos, Amazon and Netflix largely do not.
A monorepo buys you atomic cross-service changes, one dependency version, easy large-scale refactoring, and a single view of the code. It costs you tooling: at any real size you need a build system with a dependency graph and remote caching, plus test selection, or CI time becomes intolerable. You are effectively committing to funding a build team.
Polyrepo buys you independence and simple per-repo tooling. It costs you dependency drift, painful cross-cutting changes, and the situation where a security fix has to land in ninety repositories.
What I actually look at is the shape of the changes. If most changes touch one service, polyrepo is fine and cheap. If changes routinely cross service boundaries — which is common when services were split too early — a monorepo removes real friction. And I would not migrate an existing organisation without a very strong reason: a monorepo migration is a multi-year programme that consumes exactly the engineers who would otherwise be improving the platform. The pragmatic middle is a small number of repositories grouped by domain, which gets most of the atomicity without needing Bazel.
What is being tested
Whether you can hold a position without dogma, and whether you know the monorepo's real cost is a permanent build-tooling commitment.
How would you roll out a breaking change across 200 services?
What you said
A strong answer
The premise is the problem, so I'd start there: if I need 200 teams to act, I have already lost, because it will take a year and never finish. The goal is to make the change without needing 200 acts of cooperation.
The sequence I'd use. Make the new way available and better, and support both. Measure adoption from day one, publicly, per team. Migrate the long tail yourself — write the codemod, open the pull requests, and ask teams only to review and merge. The single biggest determinant of whether a migration completes is whether the platform team does the work or delegates it; delegating a low-priority migration to 200 teams means it does not happen. Then make the old way harder: a deprecation warning, then a build warning, then a build failure with a date announced well in advance. Then remove it, and hold the date.
Two things that make this survivable. Pick a small number of friendly teams first and fix what breaks before it is visible to everyone — the first three migrations always surface something you did not anticipate. And publish the deprecation date once and do not move it: a date that slips teaches everyone that the next one will slip too, and the migration after this one will be much harder.
The uncomfortable part is that some teams will genuinely be unable to move, and I'd want to know which those are early, because the plan has to accommodate them rather than discover them at the deadline.
What is being tested
Whether you do the work or delegate it. 'We'd communicate the deadline and track compliance' is the answer that fails, because it describes a migration that never finishes.
We want to put GPU training racks in our existing datacentre. Can we?
What you said
A strong answer
Almost certainly not at scale, and the reason is not space — which is the first thing I would say, because space is what people check and it is never the constraint. A hall designed for 5 to 15 kilowatts per rack has ample floor area for GPU racks and cannot power or cool them; a dense training rack draws 40 to over 100 kilowatts, so a single row can exceed what an entire aisle was designed for.
So the questions in order. What is our contracted power against our actual draw — the gap is what is genuinely available, and it is usually smaller than assumed because commitments are counted optimistically. What can the distribution physically deliver to one rack position, because even with building headroom the busway and branch circuits to that spot may cap far lower. Can we reject the heat — this is where it normally fails, since air cooling runs out somewhere around 30 kilowatts per rack, so we are talking about rear-door heat exchangers or direct-to-chip liquid and a water loop the building may not have. And can the floor carry the weight.
The honest answer is usually: we can take a small number of racks by spreading them across the hall to distribute thermal load, which wastes space and is often the right pragmatic move. Anything at scale needs a retrofit or a purpose-built facility, with lead times measured in quarters because transformers and switchgear are not quick. I would give the business the number of racks we can take today plus the cost and lead time of each option, because a refusal without alternatives just gets routed around.
What is being tested
Whether you know power distribution and heat rejection bind before space, and whether you give the business a number they can plan against.
How do you know your backups work?
What you said
A strong answer
I would answer with restore evidence rather than backup evidence, because a backup that has never been restored is a hypothesis and every organisation that has lost data had backups running successfully.
What I want: a timed restore of a real dataset, done recently, into an environment resembling production. Not a file — the actual system. And the number that matters is not whether it succeeded but how long it took, because a restore that completes in forty hours does not meet a four-hour recovery objective, and that gap gets discovered during the outage.
Then the specific failure modes I would test for. Consistency — a database copied without quiescing can restore into a corrupt state, and file-level success tells you nothing about that. Dependency order — can we restore the application if identity is also gone, because if the directory is a prerequisite and it is in scope of the incident, the plan does not work. Immutability — if ransomware holds production credentials and those credentials can delete the backups, we do not have backups; this is the modern failure that traditional plans miss entirely. And the runbook — can somebody who is not the author perform it, at three in the morning.
So my honest answer to ‘how confident are you’ is: as confident as the date of the last full restore test, and I would tell you that date. If it is more than six months old, the correct answer is that I do not know.
What is being tested
Answering with restore evidence, naming immutability against ransomware, and being willing to say 'I do not know' with a reason.
A host is slower than its identical siblings. CPU and memory look fine. Debug it.
What you said
A strong answer
‘Utilisation looks fine’ is usually where these investigations go wrong, because utilisation is an average and the causes of this are not visible in averages. The first thing I check is saturation rather than utilisation — run queue length and pressure stall information tell you whether processes are waiting, which is what slow feels like. A host at 40% CPU with a run queue of eight is saturated, and the graph says idle.
Then the asymmetries that can exist between otherwise identical machines. Thermal throttling — compare current clock against nominal, because a partially blocked heatsink or a failing fan produces exactly ‘identical hardware, one is slower’. A degraded memory channel — correctable ECC errors accumulating, or a DIMM mapped out so the host runs with less interleaving than its siblings. NUMA placement, so every memory access crosses the interconnect. A disk that has not failed but is retrying, visible in latency percentiles rather than throughput. And the network — a link negotiated at the wrong speed, or interface errors, which is embarrassingly common and invisible unless you look at the counters.
But the single most useful thing is that identical siblings exist, so I would diff everything mechanically — firmware, BIOS settings, kernel parameters, negotiated link speed, DIMM population. The difference is nearly always in that diff and finding it that way is far faster than reasoning about it. Given the symptom my first guess would be thermal or a mapped-out DIMM, because both produce this and neither shows in utilisation.
What is being tested
The saturation-versus-utilisation distinction, and reaching for a mechanical diff when identical comparators exist.
Users report the application is slow. Prove it is not the network.
What you said
A strong answer
First I would define slow, because ‘slow’ from a user and ‘slow’ on a link are different measurements. Slow to connect points at DNS or the handshake; slow to first byte points at the server or a proxy; slow throughout points at throughput or loss. That single question usually halves the search space.
Then I would measure like the application rather than like a diagnostic. ICMP is deprioritised or rate-limited on plenty of equipment, so a clean ping proves very little. I want a TCP probe to the actual port, running continuously from the affected site, giving latency and loss over time rather than a snapshot of a moment when it was fine.
Then the mechanisms that produce intermittent rather than constant slowness. Packet loss with retransmits — even a fraction of a percent destroys TCP throughput over a long path and is invisible in a utilisation graph. Micro-bursts — interface counters averaged over five minutes look idle while the buffer overflows for milliseconds, so I want discards and queue depth, not average utilisation. Asymmetric routing where the return path traverses something congested or stateful. MTU and fragmentation, which produces the classic ‘small requests work, large ones hang’. And a middlebox — firewall, proxy, TLS inspection — which is very often the actual answer and is not thought of as network.
I would get a packet capture at both ends before concluding anything. Discards and retransmits are facts; everything else is inference.
What is being tested
Rejecting ping as a measurement, and naming micro-bursts and discards over average utilisation.
An incident has recurred four times in two months, and each time we met the SLA. What now?
What you said
A strong answer
The detail that matters is that we met the SLA every time, because it means the incentives are working exactly as designed — we are measured on restoration, we restore well, and nothing forces a permanent fix. That is the thing to change, not the diagnosis.
Practically I would quantify rather than escalate. Four incidents, the engineering hours consumed each time, the customer impact, and the cumulative cost set against the estimated cost of the permanent fix. That comparison is usually stark and it is what moves a backlog item that has been deprioritised four times, whereas asking again does not.
Then I would find out why it has not been fixed, because it is rarely that nobody knows the cause. Usually the fix belongs to a team with different priorities, or it needs a change to something nobody owns, or the workaround is cheap enough that it never reaches the top of a queue. Those are three different problems needing three different escalations.
I would also stop the automatic restore being invisible. If a runbook silently fixes it at three in the morning, the organisation never feels the cost, so I would report the recurrence as one line rather than four separately-closed green incidents.
And in the meantime I would improve detection and reduce restore time — but I would be explicit that this is mitigation rather than resolution, so nobody mistakes the improved numbers for the problem going away.
What is being tested
Recognising that meeting SLA is why it has not been fixed, and separating mitigation from resolution honestly.
How would you plan capacity for a system you have never operated?
What you said
A strong answer
I would build it as a ladder with stated assumptions, and say out loud which assumption the answer is most sensitive to, because that is where I would spend effort measuring.
Start from the given — requests per day, users, transactions. Convert to peak rather than average, because systems fail at peak; real traffic typically peaks at two to three times its mean, and I would say I am assuming that. Then per-unit capacity: what one node serves at a sustainable utilisation, which I would measure if I can and state as an assumption if I cannot. Divide. That is the number most people stop at and it answers a question nobody operates. Then multiply for failure domain — survive losing a node and survive losing a zone, which is usually about 1.5×. Then multiply for headroom, because growth is real and autoscaling is not instant.
Then the part that shows experience: name the binding constraint, because it is rarely CPU. Usually it is connections, memory, IOPS, a downstream rate limit or a licence, and if I cannot name which then I have not finished the design. The corollary is that scaling the obvious tier often changes nothing because the queue simply moves.
I would also apply Little's Law — concurrency equals arrival rate times latency — because it tells me the connection pool, the thread count and the memory footprint, and it explains the death spiral where rising latency raises concurrency until resources exhaust.
What is being tested
Whether you produce a defensible number with assumptions attached, and whether you name the binding constraint.
Delivery has slowed across four teams over two quarters. Diagnose it.
What you said
A strong answer
I would start with a hypothesis set rather than open-ended listening, because forty engineers generate enough noise that you can talk to people for a quarter and learn nothing.
There are only a few things that usually cause this, and they are all measurable within about a week from existing data. Dependencies — teams cannot finish without another team, so everything blocks on coordination; I would measure what fraction of work items had a cross-team dependency and how long they waited. Work in progress — too much started, nothing finished, which shows as flat throughput with rising cycle time. Toil — the proportion of capacity going to operational load, which grows silently and nobody re-baselines. Quality debt — rising change failure rate, so more of each cycle is rework. Or the work genuinely got harder, which is a real answer and often the correct one.
Those point at different fixes. Dependencies are an org-design problem, and I would look at whether team boundaries match the architecture — if they do not, every feature crosses three teams and no amount of process fixes it. Toil is a platform investment. Work in progress is a discipline problem I can fix quickly.
I would be most suspicious of dependencies, because it is the most common and the most invisible: every individual team looks busy and productive while nothing ships. And I would be careful about velocity as a measure — it is team-relative and gameable, so I would look at cycle time and throughput of things that reached users.
What is being tested
An organisational-design answer with measurements attached, rather than open-ended listening. Scepticism about velocity is the tell.
Headquarters treats this centre as an execution arm. What would you do?
What you said
A strong answer
I would treat it as an operating-model problem rather than a cultural one, because trust-building is what everyone says and it does not work when the org chart says something different.
Three things I would check in the first month. Does anyone here own a product outcome end to end, or do we own components of outcomes owned elsewhere? If it is the second, no amount of relationship-building changes the perception. Where is the roadmap set, and is someone from this site in that room at the point of decision rather than at the readout — because being consulted is not deciding. And what is the attrition of our strongest people, and where do they go? If they leave at the same level for product companies, we are not offering scope, and that is measurable rather than a matter of opinion.
Then I would pick one charter — a whole product surface, small enough that headquarters can afford to let it go — and get it moved here with the decision rights attached, not just the headcount. One genuine transfer of ownership does more in a year than five years of relationship work, because it changes what people here can put on a CV, which is what actually retains them.
And I would want that charter agreed before taking the role rather than promised after. Scope that depends on future goodwill is the single most common reason these roles disappoint.
What is being tested
Structural diagnosis with observable evidence, and negotiating the mandate before accepting rather than after.
Tell me about someone who was underperforming.
What you said
A strong answer
The one I learned most from I got wrong first. An engineer's output had dropped over about two months and my initial read was motivation, so I did the standard thing — clearer goals, more frequent check-ins. It did not help, and in hindsight it made things worse because it added scrutiny to someone already struggling.
What I had not done was ask a direct enough question. When I eventually did — not ‘is everything okay’ but ‘something has changed and I would rather understand it than guess’ — it turned out they had moved onto a part of the system nobody had handed over properly, and had spent six weeks not asking for help because they believed they should already know it.
The fix was structural rather than personal: an explicit handover, a named person to ask, and I changed how we onboard people onto unfamiliar areas so it did not depend on someone being confident enough to admit they were lost.
What I take from it is that I now treat a sudden drop as a signal about the system or the situation until proven otherwise. I have since managed someone out and that was the right call, but I got there having genuinely ruled out the things that were mine to fix, which is what made it defensible to them and to me.
What is being tested
Diagnosis over process, a wrong first read admitted, a structural fix, and evidence you can also be decisive when it is right.
How would you raise the standard across teams you do not control?
What you said
A strong answer
I would assume I cannot compel anything and design for that from the start, because influence without authority is a design problem rather than a relationship problem.
The mechanism that actually works is making the good path the easy path. If the standard requires teams to do extra work, adoption tracks how much they like me, which is not a strategy. If it arrives as a template, a library or a pipeline stage that removes work they were already doing, adoption is close to automatic. So my first investment is always the paved road rather than the policy.
Second, make the current state visible without punishing it — a published scorecard per team showing where they stand. Visibility alone moves a surprising amount, provided it reads as information rather than a leaderboard; the moment it becomes a ranking, teams optimise the metric instead of the outcome.
Third, pick two teams who want to be first and make them visibly successful, because peer evidence beats mandate and the second adopter is always cheaper than the first.
Fourth, reserve hard gates for genuinely non-negotiable things and be sparing, because a gate spends political capital every time it fires. One or two, not fifteen.
And I would agree upfront with engineering leadership what happens when a team simply refuses, because discovering there is no answer during the first refusal is how these functions become decorative.
What is being tested
Whether you know influence without authority is a design problem, and whether you are sparing with gates.
What is the largest technical impact you have had?
What you said
A strong answer
The one I would pick is not the biggest thing I built — it is a decision I changed.
We were nine months into a migration to microservices and I was not on that team. I kept seeing the same pattern in incident reviews: outages caused by coordination between newly-split services, not by the monolith. So I spent two weeks on the analysis properly — incident causes for the previous year, deploy frequency by team, and where lead time was actually going, which turned out to be code review rather than deployment.
I wrote it up in six pages with the data and argued we were solving a problem we did not have while creating one we did. That is an uncomfortable document to circulate nine months into a funded programme.
What happened: we stopped splitting further, kept the three services with genuine independent scaling needs, and redirected the effort at review latency and test time, which was the real constraint. Lead time halved in a quarter, and the migration would have taken another year without moving it.
The part I would call Principal-level is not the analysis — a senior engineer could have done that. It is that I did it for a programme I was not on, wrote it so people could change their minds without losing face, and got a decision reversed without a fight. And I was wrong about one thing: I argued we should collapse the three back, and the team pushed back with scaling data. They were right and I said so publicly, which is probably why the rest landed.
What is being tested
Scope beyond your own team, a decision changed rather than executed, and a public admission of being wrong.
A programme will miss its date, six weeks out. What do you do?
What you said
A strong answer
The first thing I would do is tell people, the day I became confident rather than the day it became undeniable. Six weeks of warning is a decision; two weeks is an announcement, and the single biggest determinant of how badly a slip lands is when it is communicated.
Before that conversation I want three things. Why, specifically — not ‘it took longer than expected’, but was the scope wrong, was there an unforeseen dependency, did we lose people, did quality force rework? The cause determines whether a new date is credible. What is still achievable by the original date, because the useful conversation is almost never all-or-nothing; it is which subset ships and what that subset is worth. And what the new date is, with a confidence level and what would have to be true for it to hold.
What I would not do is add people. Six weeks out, onboarding cost exceeds contribution and it makes the date worse while looking like action.
Then the harder half: what I would change so this surfaces earlier next time. Usually the answer is that we were tracking output rather than remaining uncertainty, and the slip was visible in the data three weeks before anyone said it aloud. Fixing that is worth more than recovering this date.
What is being tested
Communicating early, refusing to add people, offering the subset, and naming what would have surfaced it sooner.
You inherit a backlog of forty items and three stakeholder groups. Walk me through your first month.
What you said
A strong answer
I would not start with the backlog, because a backlog is an accumulation of past requests rather than a statement of what matters now.
Three things. First, find out what the business is actually measured on this year — not the mission statement, the number the executive team reports upward. Nearly every prioritisation argument is a disagreement about that conducted without anyone naming it, and once it is explicit a surprising number of contested items resolve themselves.
Second, find the constraint. Is the business limited by demand, by delivery, or by retention? Those point at completely different quarters and forty backlog items will contain plausible work for all three. Building beautifully against the wrong constraint is the most expensive mistake available to a product manager and it is invisible at the time.
Third, go and look at where capacity actually goes. In my experience a large fraction is already committed to unplanned work — escalations, keeping the lights on, a migration somebody promised. If half the quarter is spoken for, prioritising the other half against a forty-item list is the wrong exercise; the real conversation is about the half nobody is looking at.
Then I would publish three things with what each is meant to move, and an explicit not-doing list — that second half is what makes it a decision. And I would say which of the three I expect to be wrong about and how we would find out by mid-quarter.
On frameworks: I would use one to structure the conversation, not to produce the answer. Anyone can move a reach estimate to change the ranking.
What is being tested
Naming the business metric, finding the constraint, and surfacing unplanned work -- the last of which most candidates miss entirely.
How would you choose a metric that is hard to game?
What you said
A strong answer
I would start by accepting that any single metric will be gamed, and design for that rather than hoping for a clean one.
Prefer a metric the user has to do something real to move — sign-ups are cheap, second-week usage is not. Always pair it with a counter-metric, because speed unpaired with quality, or engagement unpaired with complaints, will be bought at the expense of the thing you did not measure. Be careful with ratios, since they improve when the denominator falls — a conversion rate can rise because you lost the hard traffic. And name the lagging and the leading indicator separately, because revenue is lagging and unactionable within a quarter, and the leading indicator is what anyone can actually work on.
Then the OKR failure that is hardest to argue against: key results that are deliverables rather than outcomes. ‘Ship the new onboarding’ reintroduces output as the goal under a new name, and it is much more common than sandbagging because the deliverables are real work.
And whatever number I report, the question that should follow is ‘against what?’ — against last quarter, against the segment that did not get it, against what would have happened anyway. Volunteering the counterfactual unprompted is what makes a product manager credible, because almost nobody does it.
What is being tested
Whether you know every proxy is gameable and design counter-metrics, and whether you catch the deliverables-as-key-results trap.
A stakeholder escalates over your head. How do you handle it?
What you said
A strong answer
I would not treat it as a loss, and I would not defend the decision. What I would do is make sure the person above me has the same information I had.
If they decide differently with full information, that is legitimate — they see things I do not, and my job was to make the decision informed rather than to win it. If they decide differently because they were told something incomplete, that is worth correcting immediately and unemotionally, with the facts rather than with any complaint about being gone around.
Then I would look at why it escalated, because most escalations are not really about urgency — they are about someone not knowing where they stand. If the queue and the reasoning had been visible, half of these do not happen. So the durable fix is usually publishing the list with positions and reasons rather than handling each escalation individually.
And I would say the decision, once, in writing, somewhere they can see it. A large share of stakeholder friction is people relitigating something they were never told had been decided.
What I would avoid is saying no to the person rather than to the trade. ‘I can do that, and it means X moves to next quarter — which do you want?’ puts the decision where it belongs and is very hard to argue with. A product manager who has never been overruled is either extraordinarily persuasive or not proposing anything contested.
What is being tested
Whether you can be overruled gracefully, and whether you fix the cause rather than the instance.
How would you price a feature with variable cost per use?
What you said
A strong answer
From cost per resolved task rather than per API call, because retries, failed attempts and human escalation are real cost and pricing off the call makes the unit economics look better than they are.
Then model the distribution rather than the mean, which is the mistake that kills margins on usage-priced products. A small tail of very expensive sessions can dominate total cost while the average looks fine, so I want p50, p95 and p99 cost per account per month, and I would price against p95.
Then choose a structure that bounds the downside. Usage tiers with an included allowance and clear overage is the most honest and easiest to explain. Seat pricing with a fair-use cap works when usage correlates with headcount. Pure pass-through is transparent and loses enterprise deals, because nobody can budget for it.
Two things I would insist on regardless. Instrument cost per customer from day one, because it cannot be retrofitted and without it you will not discover the loss-making segment until it is large. And build a circuit breaker — a per-account spend cap that degrades rather than bills without limit — because the first real incident in this space is always a runaway loop on one account.
And I would say plainly that inference prices fall over time, so a thin margin today may be fine in a year. But that is a bet and it should be stated as one.
What is being tested
Whether the denominator is right, whether you price against the tail, and whether the circuit breaker exists before the incident.
Half the teams are not using your internal platform. What now?
What you said
A strong answer
Fifty percent is actually informative, because it means the product works for some real segment — so this is a segmentation question rather than a quality question.
I would find out what the non-adopters have in common, because it is rarely random. Usually one of four things. A capability gap, where they need something we genuinely do not do, and the honest answer may be that they are a legitimate exception. An unfunded migration, where they agree it is better and nobody gave them the time — in my experience the largest group and the most fixable. They predate us and their existing setup works, so switching cost exceeds benefit. Or a trust problem from an earlier platform promise that did not hold.
Each needs a different response and only one is ‘build more features’. For the unfunded-migration group, the highest-leverage thing my team can do is do the migration for them — write the change, open the pull request, ask only for a review. That converts a six-week ask into a ten-minute one, and it is the single biggest determinant of whether internal migrations complete.
I would also stop treating adoption as one number. Fifty percent of teams might be ninety percent of new services and ten percent of legacy, which is a completely different situation and might mean we are winning.
Mandate I would keep in reserve. Spending it here buys compliance from teams who then generate the support load and tell everyone the platform is bad.
What is being tested
Treating stalled adoption as segmentation, doing the migration work rather than delegating it, and disaggregating the metric.
Every team reports green. How do you know whether that is true?
What you said
A strong answer
Green status is the least reliable signal in programme management, because reporting green is the socially safe answer until it is undeniably false. So I would not rely on it, and I would say so to the teams — the goal is to make it safe to report amber early.
What I actually look at. Has any work been completed and accepted by the team downstream of it? Not ‘in progress’, not ‘ninety percent’ — percentage complete is unfalsifiable and hovers at ninety for weeks. Have the handoff dates moved since last month, because silent drift in a dependency date is the clearest leading indicator there is and it shows weeks before anyone declares a problem. Is scope moving quietly, since teams protect dates by shrinking what they deliver without saying so, and the date is then met and the thing is smaller. And is unplanned work rising, because if a team's capacity is being eaten by escalations the plan is already invalid whatever the status says.
And I ask a different question: not ‘are you on track’ but ‘what would have to go wrong for you to miss this, and how likely is it?’ People will discuss risk far more readily than they will change a status colour, because the first is a conversation and the second is an admission.
What is being tested
Rejecting percentage-complete, watching for silent date drift and quiet scope reduction, and making amber socially safe.
Upstream renamed a column and your pipeline broke. Stop it recurring.
What you said
A strong answer
Detecting it faster is worth doing and it does not solve the problem, because the change has already shipped and I am still the one paged for a decision someone else made.
The structural fix is a data contract whose check runs in the producer's pipeline, so their build goes red before merge rather than mine at three in the morning. That single detail — whose CI fails — is the whole difference, because it moves the cost to the person who can prevent it. A schema test in my own pipeline improves my detection and changes nothing about the incentive.
The contract needs schema, but the part people leave out is semantics: what the field means, its unit, its currency. More breakages come from a silent unit change — cents to dollars, seconds to milliseconds — than from a type change, and no schema validator catches that.
Adoption is political rather than technical, so I would not mandate coverage. I would start with the two or three sources that have actually caused incidents, make the check trivially cheap to add, and bring the incident record: ‘this cost eleven hours and finance was wrong for two days’ persuades where a governance principle does not.
Meanwhile, yes to detection — freshness and null-rate checks on the fields I depend on, because a contract programme takes a quarter and I need to survive this one. And I would keep the raw layer append-only and untransformed, so whatever happens I can reprocess.
What is being tested
The fork is whose build fails. Adding validation to your own pipeline improves detection; moving the check upstream changes the incentive.
Two dashboards show different revenue for the same month. Fix it.
What you said
A strong answer
First I would establish that neither is wrong, because they usually both correctly compute a different thing — and opening with ‘which is correct’ turns it into a fight between two teams.
The usual causes are a short list: refunds included or excluded, order date versus recognition date, currency converted at transaction rate versus month-end, and test accounts filtered or not. I would diff the two definitions and put both in front of the teams rather than adjudicating privately, because the disagreement is theirs to resolve.
Then the real question, which is not which number is right but who decides what revenue means. That is a business decision and my job is to make it once and then make it the only accessible implementation — one model in the core layer, owned, documented, tested.
Then the harder half, which is deprecating the two existing paths. If both remain, we are back here next quarter with two different dashboards, and the modelling work will have achieved nothing. Deprecation is the actual deliverable.
I would also find out first whether either number feeds something consequential — a board pack, a commission calculation — and tell that owner personally rather than letting them discover a number moved.
What is being tested
Recognising both are correct, escalating the definition to the business, and treating deprecation as the real work rather than the cleanup.
When would you not build a streaming pipeline?
What you said
A strong answer
Most of the time, and being able to say so is a stronger signal than enthusiasm for it. The question I ask first is: what decision gets made differently because this is seconds fresh rather than fifteen minutes fresh? For fraud, trading, operational alerting and personalisation there is a real answer. For a dashboard a human reads twice a day there is not, and micro-batch every few minutes gives the same business outcome for a fraction of the operational cost.
What you take on when you choose streaming is three genuinely hard problems that no tool removes. Late and out-of-order events — a mobile client goes offline and sends four hours later into a window you already closed and reported on, so watermarks and allowed lateness force you to decide explicitly whether to discard correct data or retroactively change a published number. Exactly-once as a property of the whole path, not the broker — a pipeline with exactly-once processing and a non-idempotent sink is at-least-once, and idempotent writes are usually simpler than transactional semantics. And state that grows, must be checkpointed, and must restore quickly or the job cannot be operated.
So I would build it where latency changes a decision, and I would be honest that a large share of systems described as real-time are micro-batch underneath and that is usually correct. The strongest version of this answer names a time I argued against streaming and was right.
What is being tested
Whether you can justify not building the more impressive thing, and whether you know the three hard problems are yours regardless of tooling.
If a customer asks us to delete their data tomorrow, what do we do?
What you said
A strong answer
I would treat the question as a lineage test rather than a legal one, because the honest answer almost always reveals copies nobody had listed.
The obvious place is the warehouse. Then the ones people forget: materialised marts and aggregates derived from it, backups and snapshots, exports sent to a vendor or a partner, an analyst's local extract, the eval or training dataset if anything ML-adjacent was built, and the logs — application logs and pipeline logs routinely contain the same records with none of the classification.
So my answer is procedural. Can we find every location for one subject — which requires a real inventory and column-level lineage, not a document. Can we delete from each within the statutory window, including from backups, where the usual approach is crypto-shredding rather than surgical deletion because you cannot rewrite a backup. And can we prove it afterwards, because an unevidenced deletion is not a defensible one.
If we cannot answer within about an hour, the finding is not the erasure request — it is that our inventory is fiction, and that has consequences well beyond privacy. Which is why I would rather run this as a drill on a synthetic subject before a real request arrives, since the first real one is a poor time to discover the fourth copy.
What is being tested
Whether you name the copies people forget — backups, logs, extracts, eval sets — and whether you treat it as an inventory test.
The warehouse bill doubled this quarter. Where do you look?
What you said
A strong answer
First I would check whether it is a problem, because doubled spend against doubled usage is not the same as doubled spend against flat usage. So the number I want is cost per unit of work — per query, per pipeline run, per active consumer — and its trend. That is a different conversation from ‘the bill went up’.
Then attribution, which is usually the real blocker. If we cannot attribute spend to a team or a pipeline, nobody owns anything and every discussion is about the total. Tagging warehouses and queries by team is unglamorous and it is the prerequisite for everything else.
Then the specific causes, roughly in order of how much they usually account for. Full refreshes where incremental would do — typically the single largest saving. Full scans on unpartitioned tables, which you can find from query history rather than guessing. Scheduled dashboard refreshes nobody needs — check last-viewed timestamps, because a meaningful share of hourly refreshes serve reports opened monthly. Idle or oversized warehouses chosen once and never revisited. And storage nobody owns — old snapshots, abandoned dev schemas, raw retained forever.
Then I would publish cost per team monthly and let them see their own number, because they know which pipelines are safe to turn off and I do not. Visibility with ownership moves more than any central optimisation programme.
What is being tested
Reframing to cost per unit of work, naming attribution as the blocker, and putting the number in front of the people who create it.
How would you build data quality that people actually act on?
What you said
A strong answer
By having far fewer tests than people expect, because the failure mode here is identical to a badly tuned SIEM: hundreds of tests, thousands of alerts, and a team that has correctly learned to ignore them — which means a real failure is now invisible inside the noise.
I would start with four checks on the models people actually depend on. Freshness, because most incidents are absence rather than corruption and this is the highest-value single test. Volume against an expected band, which catches partial loads — worse than failures, because they look successful and produce wrong numbers all day. Uniqueness on the key, which catches fan-out from an upstream change and therefore silent double-counting, the hardest bug to notice. And null rate on required fields, usually the first symptom of a broken contract.
Almost everything beyond those four is either a business rule that belongs in the model or a test somebody added after an incident and nobody revisited.
Then the metrics. I would report the proportion of incidents found by a check rather than by a human, and median time from breakage to detection. I would specifically not report test count or coverage, because both reward adding tests, which is the behaviour that produced the noise. A team with 900 tests and a 40% alert-to-action rate is worse off than one with 60 tests and 90%.
And I would invest in lineage, because at 07:00 the question is not what broke but what depends on it.
What is being tested
Whether you know test count is a vanity metric, and whether you name partial loads — which look like success — as the thing volume checks catch.
The experiment is flat but two secondary metrics moved. Do you ship?
What you said
A strong answer
First I would establish whether flat means ‘no effect’ or ‘this test could never have detected the effect we care about’, because those lead to opposite decisions and people conflate them constantly. If the minimum detectable effect at this sample size was five percent and we would have been happy with two, the test has told us almost nothing and the honest report is that it was underpowered — which is my mistake at design time, not a finding about the feature.
On the secondary metrics: with enough of them, something is always significant. If we did not name a primary metric beforehand then we do not have a result, we have a search. I would report them as exploratory and explicitly not as evidence, and if one is genuinely interesting the correct next step is a new test with that as the primary rather than shipping on it.
Then I would separate the statistics from the decision, because muddling them is what damages trust in the function. The data says we have no evidence this improves the primary metric. Whether to ship anyway is a product judgement and there are legitimate reasons — a strategic bet, reduced support load, unblocking something else. I would rather they ship with clear eyes than have me manufacture a justification.
What I would not allow into the record is ‘flat but directionally positive’, because that phrase is how organisations accumulate a portfolio of changes none of which did anything.
What is being tested
Distinguishing underpowered from null, refusing multiple-comparisons evidence, and separating the statistical claim from the product decision.
How would you measure the impact of something you cannot randomise?
What you said
A strong answer
By naming a method and, more importantly, the assumption it rests on — because a causal claim with a stated assumption is arguable and one without is a correlation with confidence.
The options, roughly in order of how often they apply. Difference-in-differences: compare the change over time in the treated group against an untreated one, which assumes both would have moved in parallel absent the change — and that parallel-trends assumption is checkable on pre-period data, so I would show that check rather than assert it. Regression discontinuity where a threshold exists, comparing just either side of it, which is the cleanest of these when a real cutoff exists. Synthetic control, constructing a comparator from a weighted blend of untreated units, useful when there is no single good control. And instrumental variables, which is the most fragile because a genuinely valid instrument is rare and usually the hardest part of the analysis.
Whichever I use, I would be explicit that the estimate is conditional on the assumption, and I would look for what would falsify it — a placebo test on a period where nothing happened is cheap and it is the thing that separates a defensible causal claim from a plausible one.
And I would say plainly when none of these work. Sometimes the honest answer is that we cannot establish causation here and should stop presenting the correlation as though we had.
What is being tested
Whether you name the assumption rather than the technique, and whether you run a placebo or parallel-trends check rather than asserting.
We run both Databricks and Snowflake. What is your plan?
What you said
A strong answer
My default assumption is that consolidation is the wrong first move, because migrations of this size stall and you end up carrying both plus a half-finished migration — which is worse than either and is usually the situation people are already in when they ask this.
So the first question is why we have both. If it is an acquisition, or two org units that each chose correctly for their workload, then coexistence is a legitimate architecture and the work is to make it cheap rather than to end it. If it is drift with no owner, that is a different conversation.
What I would do regardless, and quickly: one storage layer in an open table format and one place where conformed entities are defined, with both engines reading it rather than each holding its own copy. That removes the failure that actually costs money — two customer tables with two definitions of active, so every number in the business depends on which engine you happened to query. Iceberg and Delta interoperability make that genuinely possible now, which was not true three years ago.
Then attribution before optimisation, because I would expect that today nobody can say what each platform costs per team, which means nobody owns it and every conversation is about the total.
And only if consolidation still looks right, I would fund the finish before starting — including the long tail — with a decommission date I intend to hold. I would also be honest with the board that the saving is real but slower than the business case will claim, because these cases always assume the last thirty percent happens.
What is being tested
Refusing the obvious consolidation answer, naming the duplicate-definition failure as the real cost, and being honest about migration economics.
Databricks or Snowflake — which would you choose?
What you said
A strong answer
I would push back on the framing before answering, because they started at opposite ends and have converged, so ‘which is better’ is largely an out-of-date question. Both now do SQL, notebooks, ML, streaming, open table formats and governance.
What still differs is the centre of gravity. Databricks assumes engineers who write code and want control — files in object storage, clusters you size and tune, strongest for ML, streaming and heavy unstructured transformation, and the cost of that is that cluster tuning is real work and idle clusters are real money. Snowflake assumes analysts who write SQL and want no operations — strongest on concurrency, ease of operation and sharing, and the cost is that a runaway query is very cheap to write and expensive to run, and escaping SQL for heavy engineering is less natural.
So my answer depends on who the primary user is. If the team is engineers building pipelines and models, Databricks. If it is a large analyst population who need self-service SQL with minimal operations, Snowflake.
But I would add that in most enterprises the real question is not which to pick — roughly two-thirds of enterprise postings name one of them and a third name both. Running both is usually an acquisition or two units that each chose well, and the work is making them coexist over one storage layer and one set of conformed definitions rather than picking a winner.
What is being tested
Whether you know the convergence has made the comparison stale, and whether you can name the real enterprise question, which is coexistence.
Walk me through what happens when I type a URL and press enter.
What you said
A strong answer
I'd give you the eight-step version and then tell you which of them actually break, because the second part is the useful half.
One, the name becomes a number. Browser cache, then OS cache, then /etc/hosts, then the recursive resolver, which walks root → .com → the authoritative server. Two, is it local? The machine masks the destination against its own subnet. It almost never is, so it goes to the default gateway — and to send anything to the gateway it needs the gateway's MAC, so it ARPs for it. Three, the switch looks up the destination MAC and forwards, flooding if it does not know it. Four, the router does longest prefix match, and on the way out most networks NAT the source address and keep a translation table entry. Five, ten to twenty hops of other people's networks, with the path chosen by BGP — which optimises for policy, not for speed. Six, TCP handshake. One round trip. Seven, TLS. One more in 1.3, two in 1.2. Eight, the load balancer picks a backend and the application finally runs.
Now the part that matters. On a Mumbai-to-Virginia path at 180 ms, an uncached DNS lookup plus TCP plus TLS plus the request is four round trips — about 720 ms before your code has executed a single line. That is the entire commercial case for CDNs, and it is about moving handshakes closer rather than about bandwidth.
And the three that actually break in production: DNS TTL, because it is a cache you cannot purge; MTU, when a tunnel is added and ICMP is blocked, which gives you the signature where small requests work and large ones hang; and certificate expiry, which is still the most common self-inflicted outage in the industry.
What is being tested
Whether you can name the mechanisms and then say which of them actually break. Most candidates recite four steps and stop.
Split 192.168.20.0/24 into subnets for 6 teams of up to 25 hosts each. Show your working.
What you said
A strong answer
Six subnets means I need to borrow enough bits that 2^n is at least 6, so n = 3, giving eight subnets and a /27 mask — 255.255.255.224.
A /27 has a block size of 32, so 32 addresses per subnet, minus one for the network address and one for the broadcast, leaves 30 usable. That covers 25 with headroom.
The boundaries fall on multiples of 32:
Team 1: 192.168.20.0/27 — hosts .1–.30, broadcast .31
Team 2: 192.168.20.32/27 — hosts .33–.62, broadcast .63
Team 3: 192.168.20.64/27 — hosts .65–.94, broadcast .95
Team 4: 192.168.20.96/27 — hosts .97–.126, broadcast .127
Team 5: 192.168.20.128/27 — hosts .129–.158, broadcast .159
Team 6: 192.168.20.160/27 — hosts .161–.190, broadcast .191
Two spare: .192/27 and .224/27.
Two things I would flag rather than leave implicit. First, the gateway is not .1 for five of these six subnets — it is the first or last usable address of each block, so .33, .65, .97 and so on. Hard-coding .1 as the gateway is a real and common production bug and it fails silently for everyone except team 1.
Second, if the requirement had been 35 hosts per team instead of 25, this would be impossible in a single /24: /27 gives only 30 hosts, and /26 gives 62 hosts but only four subnets. I would need a second /24, or a /23 to begin with. Noticing when the constraints are incompatible is usually the actual point of the question.
What is being tested
Arithmetic you cannot bluff — plus whether you notice the gateway is not .1 for five of the six subnets.
Small API requests work. Anything over about 2 MB hangs and times out. Diagnose it.
What you said
A strong answer
Size-dependent failure with a clean cutoff, where small works and large hangs rather than errors, is nearly always path MTU discovery being black-holed. I would say that up front and then prove it rather than guess further.
The mechanism. Every link has a maximum frame size, normally 1500 bytes. When a packet is too big for a link and has the don't-fragment bit set, the router is supposed to return an ICMP type 3 code 4 — ‘fragmentation needed’ — so the sender reduces its segment size. A great many corporate firewalls block all ICMP on the theory that ICMP is a security risk. So the message never arrives, the sender keeps sending 1500-byte packets, they keep being silently discarded, and TCP retransmits forever. Small requests fit in one segment and are completely unaffected, which is why the symptom looks so strange.
Proving it takes one command. ping -M do -s 1472 <host> sends a 1500-byte packet with don't-fragment set. If a plain ping succeeds and that one fails, it is confirmed. Then bisect the size down — 1400, 1300 — to find the real path MTU.
What usually caused it. Something added a tunnel. A new VPN, a VXLAN overlay, a PPPoE link, a cloud VPN gateway — each adds encapsulation overhead and reduces the effective MTU below 1500. The bug report never mentions the tunnel, because whoever added it did not connect the two events.
Fixes, best first. Stop dropping ICMP fragmentation-needed — it is a required part of IP, not an optional diagnostic. If the firewall team will not move, clamp MSS at the tunnel endpoint so TCP negotiates a smaller segment size at handshake time, which works without any ICMP at all. Lowering the interface MTU everywhere is the blunt instrument and it costs you throughput on paths that did not need it.
And the prevention: add an MTU check to whatever you run after any tunnel or overlay change. This class of bug is entirely predictable and almost never tested for.
What is being tested
Pattern recognition. Size-dependent failure has one dominant cause, and naming it immediately is the whole signal.
About 1% of requests return 502. Application logs show nothing. Where do you look?
What you said
A strong answer
The fact that the application logs show nothing is the most useful clue in the question, because a 502 is generated by the proxy — it means the proxy could not get a valid response from a backend. If the application never logged the request, it never received it. So the proxy's access log is the source of truth, not the app's.
First, split on upstream response time. If it is zero, the proxy could not establish a connection at all. If it is exactly your timeout value, the backend accepted the connection and never answered. Those are completely different bugs and I would not proceed until I knew which.
Then correlate. Do the 502s cluster on one backend instance? Then it is that instance and I look at its own resources. Are they even across the fleet? Systemic. Do they spike at deploy times? That is connection draining — old pods stopped listening before the load balancer stopped routing to them — and the fix is a preStop delay plus graceful shutdown.
The one most people miss, and the one I would check early because it fits ‘steady sub-percent with no pattern’ precisely: keep-alive timeout mismatch. If the proxy holds idle upstream connections for 75 seconds and the backend closes them at 60, then in that 15-second window the proxy will occasionally reuse a connection the backend has just closed. The request fails with no application-side trace whatsoever. The rule is that the backend's idle timeout must always be comfortably longer than the proxy's, and this single misconfiguration explains a surprising number of unexplained low-rate error rates.
Also worth checking: conntrack table exhaustion on any NAT device in the path — dmesg will say so explicitly — and ephemeral port exhaustion on the proxy if it opens many short-lived outbound connections.
And I would treat 1% as worth fixing rather than tolerating. A persistent 1% is roughly one bad experience per hundred, which at any real volume is a lot of people, and it is usually a single configuration line.
What is being tested
Whether you know that a 502 comes from the proxy, so the app's silence is evidence rather than a dead end.
We are choosing between colocation and cloud for a steady workload. How do you decide?
What you said
A strong answer
I would refuse to answer it as a binary, because the honest answer is that they win on different properties and the workload decides — and then I would actually do the arithmetic, because that is what makes the recommendation credible.
What cloud is genuinely selling. Elasticity, and time to first server. If load is spiky, seasonal or unknown, you are buying the ability to not own the peak, and that is worth a large premium. If you need a server this afternoon, there is no colocation answer.
Where colocation wins decisively. Two places. Bulk egress — cloud egress is priced as a product, not a cost pass-through, and wholesale IP transit is an order of magnitude cheaper, so a high-egress workload can be dominated by this single line. And sustained accelerator capacity — renting GPUs continuously for three years can cost several times what buying them costs, which is exactly why several large AI companies moved to owning.
The method. Build a three-year total for both, and include the things people leave out of the colocation side: hardware refresh, spares, remote hands, a network engineer, and the fact that you bought the peak rather than the average. Include the things people leave out of the cloud side too: egress, cross-AZ traffic, NAT gateway processing, and support tier.
What the colocation bill is actually made of, so the estimate is real: rack rental, committed kW (which is the main driver — you rent power, and space comes with it), metered overage, cross-connects per month per connection, IP transit billed on the 95th percentile, remote hands per hour, and one-off install charges.
Where I would land in most cases: split it. Elastic, new and unpredictable things in cloud; large steady-state things with heavy egress or continuous GPU load where you can buy them outright. And be explicit about what you are giving up — the moment you own hardware, you own capacity planning, and a bad forecast is now a purchase order rather than an API call.
What is being tested
Whether you can do the arithmetic out loud, and whether you resist answering a false binary.
Explain security groups versus network ACLs, and when each is the wrong choice.
What you said
A strong answer
The distinction is stateful versus stateless, and everything else follows from it.
Security groups are stateful and allow-only. They attach to an instance or network interface. If you allow inbound on 443, the response is automatically permitted out — the connection is tracked. There is no deny rule; the absence of an allow is the deny, which means you cannot express an exception.
Network ACLs are stateless and support both allow and deny. They attach to a subnet, so they apply to everything in it. Because they are stateless, allowing inbound 443 is not enough — you must separately allow outbound on the ephemeral port range, 1024–65535, or the response is dropped. This is why a NACL change so often breaks something that looks entirely unrelated: someone tightened outbound rules and killed the return path of inbound connections.
Where each is wrong. A NACL is the wrong tool for ordinary application access control — it is coarse, it is stateless, and it is subnet-wide, so it cannot express ‘this service may reach that service’. And a security group is the wrong tool for blocking a specific known-bad range, because it has no deny.
The practice worth adopting is referencing security groups from other security groups rather than IP ranges. Write the rule as ‘the web tier security group may reach the database tier security group on 5432’ and it stays correct forever as instances come and go. Rules written against CIDR blocks rot the moment the topology changes, and nobody ever removes them, so estates accumulate hundreds of stale allow rules that nobody dares delete.
What is being tested
The single most common cloud networking question. Stateful versus stateless, and the ephemeral-port consequence.
How would you make DNS resilient for a company whose revenue depends on it?
What you said
A strong answer
I would start by naming why DNS is uniquely dangerous, because that shapes every decision: it is a globally distributed cache with time-based invalidation and no purge mechanism. Every change you make propagates on a schedule you set in the past, to caches you do not control. Nothing else in the stack has that property.
One: a second, independent provider. This is the single highest-value change and it is cheap. Secondary DNS with a different vendor, on different infrastructure, with zone transfer or dual-push automation. The Dyn attack in 2016 took out a long list of major sites simultaneously not because they were attacked but because they had all delegated to the same provider. Your DNS vendor is a single point of failure for your entire company.
Two: TTL discipline. Short TTLs on anything you might need to move in an emergency; longer TTLs elsewhere, because a 60-second TTL on a high-traffic name is real load and real cold-client latency. And a runbook rule: lower TTLs at least 48 hours before any planned migration, so the old long TTL has expired everywhere before you need the short one.
Three: no circular dependency. This is the Facebook 2021 lesson and it is the one people skip. Their DNS servers were healthy; a BGP withdrawal made them unreachable, every name failed globally, and because badge access and internal tooling resolved through the same names, engineers could not physically get into the buildings. So: does your recovery path depend on the thing that is broken? Out-of-band access, a bastion reachable by IP, and credentials that do not require resolving a name you own.
Four: monitor from outside. Resolution checks from multiple external vantage points, not from inside your own network where your resolver may be caching a correct answer that the world cannot get. Alert on the answer being wrong, not just on the query succeeding.
Five: if you run DNSSEC, treat signature expiry as a first-class alert. A failed key rollover does not degrade your domain, it deletes it for every validating resolver. Several national-scale outages have been exactly that.
What is being tested
Whether you name the circular-dependency risk. Almost nobody does, and it is what turns an outage into a multi-hour one.
Walk me through the OAuth authorization code flow with PKCE, and tell me what each parameter defends against.
What you said
A strong answer
I'll do the flow first and then the parameters, because the parameters are where the actual security is.
The flow. The user clicks sign in. The client generates a random verifier, hashes it with SHA-256 into a challenge, and redirects the browser to the authorization server carrying client_id, redirect_uri, scope, state and code_challenge. The authorization server authenticates the user and gets consent, then redirects back to the registered redirect_uri with a short-lived code and the same state. The client then makes a back-channel POST to the token endpoint with the code, the original code_verifier, and its client secret if it has one. It gets back an access_token, an id_token and usually a refresh_token.
Now the parameters.state defends against login CSRF. It binds the callback to the session that started the flow. Without it, an attacker can start their own flow, capture their own code, and feed it to your callback while you are browsing — your session silently becomes bound to their account, and everything you do afterwards is visible to them. People dismiss this as harmless because no credential leaks. It is not harmless.code_challenge / code_verifier — PKCE — defends against authorization code interception. If an attacker grabs the code, they cannot exchange it, because the token endpoint demands the verifier and it never left the client. Originally designed for mobile apps where custom URL schemes can be hijacked; now recommended for confidential clients too, because it protects a different thing than the client secret does — the secret assumes it hasn't leaked, and secrets leak.redirect_uri must be exact-matched against a registered allowlist. Prefix or wildcard matching is the single most common OAuth implementation flaw — an attacker registers something that passes your check and receives the code.nonce, in the OIDC case, binds the resulting id_token to your specific authentication request, so a replayed token from elsewhere fails.
And the thing I would say last because it is the one most often wrong: the access token is opaque to the client. Even when it happens to be a JWT you could decode, don't. It is addressed to the API, its format can change without notice, and the id_token exists precisely to tell the client who the user is.
What is being tested
Whether you understand the parameters as defences rather than as boilerplate. Most candidates can recite the flow and cannot say what state is for.
What must an API check on every JWT it receives?
What you said
A strong answer
Six things, and skipping any one of them is a complete bypass rather than a weakness.
One, the signature — against the issuer's published JWKS, fetched over TLS and cached with key rotation in mind. And critically, pin the expected algorithm. Never read the algorithm from the token's own header. That is exactly how the two classic attacks work: alg: none, where the token declares itself unsigned and a naive library accepts it, and RS256-to-HS256 confusion, where an attacker signs with your public key used as an HMAC secret. Both are live in the wild because both libraries and developers get this wrong.
Two, iss — exact string comparison against your expected issuer. Not a prefix match, not a contains check.
Three, aud — that this token was issued for this API. This is the check most often omitted, and omitting it is serious: a perfectly valid token minted for any other service in the same tenant will be accepted by yours. That is a ready-made lateral-movement primitive, and it turns a low-privilege service compromise into access to a high-privilege one.
Four, exp and nbf — with minimal clock skew tolerance. Sixty seconds, not an hour.
Five, the scopes actually authorise the operation. A valid token is not an authorisation decision; it is one input to one. This is where a lot of otherwise-correct implementations stop, and it is why so many APIs are authenticated but not authorised.
Six, for an id_token, the nonce against what you sent.
Two practical notes. Use a maintained library rather than writing this; every item on that list has been a CVE somewhere. And if you need immediate revocation, a JWT alone cannot give it to you — you need short lifetimes plus either introspection or a revocation list, and that trade-off between statelessness and revocability should be a deliberate decision rather than something you discover during an incident.
What is being tested
The audience check. It is the one most often missing and the one with the worst consequence, and mentioning it unprompted is a strong signal.
Explain Kerberoasting and how you would defend against it.
What you said
A strong answer
Kerberoasting abuses a feature of Kerberos rather than a flaw, which is why the defence is not what people first reach for.
The mechanism. Services in Active Directory are identified by a service principal name registered against the account the service runs as. Any authenticated domain user may request a service ticket for any SPN — that is how the protocol is designed to work, because the whole point is that the service validates the ticket itself without contacting the domain controller. The ticket is encrypted with a key derived from the service account's password. So an attacker with any low-privilege domain account requests tickets for every SPN they can enumerate, takes them offline, and cracks them at their leisure. A cracked ticket yields the service account's password — and service accounts are frequently over-privileged, sometimes domain admin.
Why detection is a weak defence. The request is indistinguishable from legitimate use. You can alert on unusual volume or on requests for many SPNs from one account in a short window, and that is worth having, but a patient attacker requesting five tickets over a week will not trip it.
The real defence is making cracking infeasible. Run services as Group Managed Service Accounts, which have 120-character randomly generated passwords rotated automatically by the domain. A GMSA ticket is not crackable in any meaningful timeframe. Where a GMSA genuinely isn't possible, the fallback is a very long random password with a rotation process.
And disable RC4. RC4-encrypted tickets crack far faster than AES. Forcing AES raises the cost substantially, and the reason RC4 is still enabled almost everywhere is a legacy application nobody owns — finding those is the actual work.
Then reduce the blast radius: audit which service accounts hold privileged group membership, because there is rarely a good reason for a SQL service account to be a domain admin, and that is what turns a cracked password into a domain compromise.
The wider point I'd make is that this is the shape of most AD attacks — abusing designed behaviour rather than exploiting a bug — so the defences are configuration and hygiene rather than patching.
What is being tested
Whether you know that detection is weak here and the defence is uncrackable credentials. Candidates who say 'monitor for it' have read about it rather than defended against it.
Why is WebAuthn phishing-resistant when app-based MFA is not?
What you said
A strong answer
Because of where the origin comes from, and that single difference is a change in kind rather than in degree.
What every prior factor has in common. A password, a TOTP code, an SMS code, a push approval — all of them are things the user can be persuaded to supply to the wrong party. A modern phishing kit sits as a real-time proxy between the user and the genuine site: the user types their password into the attacker's page, the attacker forwards it, the real site asks for the OTP, the attacker asks the user, the user supplies it, and the attacker completes the login and steals the resulting session. Every factor in that list is defeated, including push approval, which is defeated by fatigue or by simply forwarding the prompt.
What WebAuthn does differently. At registration the authenticator generates a key pair bound to that specific origin, and the private key never leaves the device. At authentication, the browser — not the user, not the page — inserts the actual origin into the data that gets signed.
So on rnicrosoft.com, one of three things happens and all three are safe. Either the authenticator has no credential for that origin and there is nothing to offer. Or, if the attacker relays a challenge from the real site, the authenticator signs over rnicrosoft.com, because that is what the browser reports, and the real server rejects it. There is no secret for the user to type and no code for a proxy to forward.
Now the honest part, which I think matters more than the mechanism. Strengthening authentication moves the attack rather than ending it. As phishing-resistant methods spread, attackers go to what's left: session-token theft after authentication, consent phishing for OAuth scopes, device-enrolment abuse, and — the one that has featured in several major intrusions — the help desk. Several large companies were breached through a social-engineered credential reset precisely because the authentication itself was too strong to attack directly.
Which is why the account-recovery path is the whole programme. If you deploy passkeys and leave recovery on an emailed link or an unverified helpdesk call, you have moved the front door and left a window open. Two enrolled authenticators from day one, and a high-assurance recovery process, are what make the deployment real.
What is being tested
The origin-binding mechanism, and then whether you know that it displaces the attack to sessions and the help desk rather than eliminating it.
We are choosing an identity stack for 5,000 employees. How do you approach it?
What you said
A strong answer
I'd start by refusing to treat identity as one purchase, because it is four adjacent markets with different buyers and different economics, and conflating them is how organisations end up paying twice for the same capability.
The four: workforce access management, customer identity, identity governance, and privileged access plus secrets. Different vendors lead each, and the bundles overlap in confusing ways.
The first question is not which vendor, it's what we already own. If we are on Microsoft 365 E5, Entra ID P2 is already paid for. The marginal decision is then Entra ID Governance — and I'd flag the trap that it is an add-on, not an upgrade, so it sits on top of the $7 or $10 rather than replacing it. That single misreading turns a $7 line into $14 in a budget.
The second question is where the real gap is, and for a Microsoft-centric shop it is almost always privileged access. Entra P2's PIM covers Entra and Azure roles only — not Linux servers, not Oracle, not network gear, not the mainframe. So a Microsoft-centric stack still needs a PAM purchase, and pretending otherwise is how audits fail.
Then I'd cost the alternatives honestly. Okta Core Essentials at $14 per user now bundles lifecycle management, access governance and privileged access — that's $840,000 a year at 5,000 seats, which is more than a thin Okta tier plus SailPoint plus a dedicated PAM vendor would cost, while delivering weaker governance and weaker PAM than the specialists. What you're buying is one contract, one renewal and one integration surface. Whether that's worth the premium is a real question, not an obvious one.
Then the regulatory constraint, which frequently decides it. If this is an Indian entity under RBI or SEBI, or subject to DPDP Rule 6, the requirement text names MFA for privileged users and one-year log retention — which is, almost word for word, a session-recording PAM requirement, plus periodic access certification. That rules out the cheap open-source stack immediately: Keycloak and OpenBao have no access certification, no segregation-of-duties analysis and no attestation reporting, so a stack that looks like it costs a fifth as much is only cheap because it is incomplete.
Where I'd land: lead with what the compliance obligation actually names, then buy the smallest set of products that satisfies it, then optimise contract count. And I'd say plainly that the list price is the least interesting number in the comparison — the differences that matter are renewal leverage, integration surface, and which capability you're quietly paying two vendors for.
What is being tested
Whether you segment the market before shopping, whether you know PIM's scope limit, and whether you let the compliance text drive the requirement rather than the vendor deck.
Is technical analysis valid?
What you said
A strong answer
It isn't one claim, so I'd refuse to answer it as one — and I think that refusal is the actual answer. It's a bundle of hypotheses with wildly different evidential support, and treating them as a single thing is what makes the argument unproductive on both sides.
What has strong support. Cross-sectional momentum — buying recent relative winners — is one of the most replicated anomalies in finance, documented across three decades, dozens of countries including India, and multiple asset classes. Time-series trend following has a comparable record, and unusually it has decades of live audited track record rather than only backtests. Volatility clustering is close to a stylised fact.
What has weak support. Candlestick patterns as standalone signals. Chart patterns like head-and-shoulders — largely because identification is subjective, so different automated implementations find different instances on the same data, which makes them untestable rather than merely unproven. Fibonacci ratios specifically: price retraces, obviously, but that 61.8% beats 60% or 65% has never survived a proper test.
And the question that decides most of it: a pattern that precedes a rise 60% of the time is worthless if price rises 60% of the time anyway. Almost nobody publishing a pattern's 'success rate' publishes the base rate it should be compared against. Compared to what is the whole discipline in three words.
Why professionals still use charts, which is the part sceptics get wrong. Three honest reasons, none of which is prediction. A chart is a compression of a great deal of data. Obvious levels are genuinely self-fulfilling — if enough participants watch the prior day's high, orders concentrate there, and that mechanism doesn't require the level to have independent significance. And most importantly, risk management needs non-arbitrary reference points: a defensible place to put a stop and a volatility measure to size against. A trader with a mediocre entry rule and excellent stop placement outperforms the reverse, reliably.
So: the entries are the weakest part and get all the attention; the sizing and exits are where the expectancy lives and get almost none.
What is being tested
Whether you answer per-claim with the evidence attached, and whether you can state the control problem in one sentence.
How would you test whether a candlestick pattern works?
What you said
A strong answer
Five steps, and the second one is where almost everyone goes wrong.
1. State it mechanically. Bullish engulfing: candle 1 bearish with body at least 80% of its range; candle 2 bullish, same body ratio; candle 2's body fully containing candle 1's body. And critically, the context clause needs a definition too — 'after a decline' has to become 'the close is below the close three bars ago' or 'price has fallen at least 1 ATR'. Most published pattern statistics never specify the context, which alone makes them unreproducible, and the context choice changes the result far more than the candle rules do.
2. Build a matched control — and match it to the entry. This is the step that decides whether anything else means anything. A bullish engulfing occurs after a decline. So if I compare it against buying at random closes, I'm measuring short-horizon mean reversion and crediting it to the pattern. The control has to be a similar-sized decline without the pattern. Same instrument, same period, same number of trades, same holding period. And I'd average at least five random draws, because one draw is under-powered and will land high or low by chance and then mislead me in whichever direction it fell.
3. Report expectancy, not hit rate. (win% × avg win) − (loss% × avg loss), in R, net of realistic costs — for India that means STT on the right leg, exchange charges, stamp duty, half the spread each way, and a size-dependent impact term. A 70% hit rate with a 0.3R average win against 1R losses is −0.09R per trade. Hit rate on its own carries no information about whether something makes money.
4. Split by year. Every year separately. An edge that lives in one eighteen-month window is that window, and an aggregate number hides it.
5. Vary everything. Different context definitions, different holding periods, 2× and 5× the size. If it only works at one parameter setting, it's fitted rather than found.
What I'd expect to find, honestly: single- and two-candle patterns usually show small effects that mostly vanish against a matched control and after costs. Some survive in specific contexts — at a defined level, in the direction of a higher-timeframe trend. Which points at what they're actually good for: not the edge, but the trigger and the stop reference. The candle gives you a specific bar to act on and a specific low to put a stop under, and that's a real contribution, because the alternative is entering at a price you chose by feel.
What is being tested
The matched control. A dip entry needs a dip control, and candidates who compare to random closes are measuring mean reversion and calling it a pattern.
What matters most in building a trading system?
What you said
A strong answer
Almost the reverse of where people spend their time, which is why this is worth being explicit about.
1. Position sizing. Two traders with the identical signal and different sizing have unrecognisably different outcomes. The default that works is volatility-scaled: risk a fixed fraction of current capital, and let ATR determine the quantity. size = (capital × risk%) ÷ (ATR multiple × ATR). So ₹10 lakh capital risking 1%, stop at 2×ATR, ATR of ₹25, gives a ₹50 stop distance and therefore 200 shares. A stock with a ₹5 ATR gets 1,000 shares for the same ₹10,000 of risk. Without that, your P&L is determined by which instruments happened to be volatile rather than by your analysis.
And the corollary people miss: correlated positions are one position. Six longs in the same sector at 1% each is a 6% position wearing a diversification costume. Risk has to be capped by correlated group, not only per trade — that's the most common way a well-designed system still blows up.
2. The exit. This is where most of the expectancy lives and it's usually specified vaguely while the entry is specified precisely. The initial stop goes where the idea is wrong, structurally — not at an amount you're comfortable losing. Those are different, and conflating them gives you stops that get hit by noise. Structure first, then size the position so that stop costs an acceptable amount; if it costs too much, the trade is too big, not the stop too tight.
A fixed target caps winners, which is fine for mean reversion and fatal for trend following, where a few very large winners carry everything. And moving to break-even early feels prudent and usually converts winners into scratches — it's a testable question and the answer is often the opposite of the instinct.
3. The regime filter. Deciding when not to trade is usually worth more than improving entries, and it's more robust because it has fewer parameters. A 200-day trend filter, a volatility-regime filter, a liquidity threshold that also solves half your slippage problem.
4. Then the entry, which matters but far less than the three above.
5. And parameter values should barely matter. If the result flips when you change SMA to EMA, or the lookback from 3 to 5, you have a fragile fit rather than a finding — and testing that explicitly is a cheap, very informative robustness check.
The last thing I'd add: decide in advance what failure looks like. A drawdown beyond the worst in the backtest, or rolling expectancy below zero over n trades. Write it down before you're in the drawdown, because in the middle of one the decision gets made by the part of you that wants the pain to stop.
What is being tested
Whether you rank sizing and exits above entries, and whether you catch that correlated positions aggregate.
How would you debug 'the application is slow' before opening a single log file?
What you said
A strong answer
One command, and it settles the argument in about ten seconds.curl -o /dev/null -s -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://host/path
That decomposes a request into its phases, and each phase points somewhere completely different.
Slow name lookup → DNS. Check whether the resolver is answering slowly, or whether a search-domain suffix is causing multiple queries, which is a classic in Kubernetes with ndots:5.
Slow connect → the network or the listener. TCP handshake time is essentially round-trip time, so if it's high the path is long or congested. If it hangs entirely, that's a firewall dropping silently, whereas connection refused means a host answered and nothing was listening — which is much better news, because it means routing works and the search space is small.
Slow app-connect → TLS. Extra round trips from TLS 1.2 instead of 1.3, or the server fetching an OCSP response inline.
Fast connect and slow time-to-first-byte → the application, and the network is exonerated. This is the single most useful discrimination in the whole exercise, because 'the network is slow' is the default accusation and it is usually wrong.
Then I'd bisect rather than sweep. If one office reports it and others don't, run the same command from a mobile hotspot in the same building. If the hotspot is fast, I've eliminated the application, the region and the destination in ninety seconds, and the problem is the office network or its upstream.
The one caution on the next tool. If I go to mtr, I only care about loss that starts at a hop and persists to the destination. Loss shown at an intermediate hop with clean hops after it is that router deprioritising ICMP to its own control plane — it is not your problem, and chasing it wastes a great deal of time.
Most people start at the application logs and work down. Starting in the middle and eliminating half the stack per step is why the same diagnosis takes four minutes instead of an hour.
What is being tested
Bisection as a method, and knowing that fast-connect-slow-TTFB exonerates the network. The mtr caveat is a strong secondary signal.
How would you eliminate broken access control rather than patch it?
What you said
A strong answer
I'd stop treating it as something engineers must remember and make it structurally impossible to write the unsafe query.
Why patching fails. Broken access control is consistently the most common serious finding, and it persists not because people don't know about it but because the check has to happen at every access path. One new endpoint written on a Friday reintroduces it, and code review does not reliably catch a missing negative.
The structural fix. Push the scope into the data access layer. Either a repository whose every method takes the acting principal and injects the tenant predicate, or row-level security enforced in the database so a query without the predicate returns nothing at all. The goal is that a developer cannot express the unsafe query, rather than that they remember not to.
Then — and this is the part I'd insist on — verify the guard rather than the behaviour. Write a test that asserts a cross-tenant read is refused. Then delete the guard and confirm the test goes red.
I'd push on that because isolation tests pass vacuously with alarming frequency. If your test asks for a random UUID belonging to another tenant, it returns nothing whether or not the filter exists — the UUID partitions the data on its own. The test is green and it is testing nothing. Mutation-testing the guard is the only reliable way to know.
Two more controls I'd add. Use non-enumerable identifiers, because sequential integers make the attack trivial to automate — that's defence in depth, not a fix. And log access with the acting identity and the object's owner, so that if it does happen you can answer 'what did they reach' without guessing.
The reason I'd frame it this way in a design review is that 'we'll be careful' is not a control, and 'the linter/repository/database won't let you' is.
What is being tested
Whether you reach for a structural fix, and whether you know that isolation tests frequently pass for the wrong reason.
Where would you start on a cloud estate you have just inherited?
What you said
A strong answer
One exercise, and it can be done in an afternoon: for every role attached to something internet-reachable, compare the permissions granted against the permissions actually used in the last ninety days. Every major provider has an analyser for this.
The gap is routinely 95% or more, and it is your blast radius. Then ask one question of the biggest gaps: if these credentials were stolen right now, what could the attacker do? You will usually find a web front end that can read every bucket in the account, and narrowing that is a small change with an enormous effect.
I'd start there rather than with a scanner because cloud breaches are overwhelmingly not provider exploits — they are customer misconfiguration, and the same small set of findings recurs. Identity is the perimeter now: the control plane is a public API, so a credential works from anywhere and the high-value target is a role rather than a server.
Then, in order. Static long-lived keys — replace with workload identity federation and short-lived role assumption, which removes the class rather than rotating it. Organisation-level guardrails: public access blocks and service control policies, so that a manual console action cannot create the bad state rather than being detected four hours later. Then privilege escalation paths — a role that cannot read data but can create a role, attach a policy, or pass a role to a compute service is an administrator with extra steps, and there are open tools that enumerate these against your own account.
Then the logging question, which I'd ask early because the answer takes time to fix. Are control-plane audit logs on in every region, including the ones we don't use — attackers work in unused regions precisely because nobody looks there. Are data-plane access logs on, because those are the ones that answer 'what did they actually read', and without them you may be obliged to assume the worst. Are logs written somewhere the workload can't delete them. And is retention longer than typical dwell time, because thirty days often means the start of the intrusion has already aged out.
And I'd test it — take a question I'd need to answer during an incident and try to answer it from today's logs. Most teams discover in that exercise that they can't, and that discovery is worth more than the scan.
What is being tested
Whether you start with identity rather than with a scanner, and whether you think about the incident you will have before you have it.
Our inference bill is too high. Where do you start?
What you said
A strong answer
I'd start with one calculation rather than a list of optimisations, because it tells you which optimisations can possibly help.
The roofline. Peak compute divided by memory bandwidth gives the ridge point — on an H100 that's roughly 989 TFLOP/s over 3.35 TB/s, about 295 FLOP per byte. Prefill sits well right of that and is compute-bound. Decode at low batch sits at roughly 1 FLOP per byte and is memory-bandwidth-bound by a factor of several hundred. They are effectively two different workloads sharing a GPU.
Everything follows. Continuous batching — because decode wastes bandwidth per sequence, so more concurrent sequences amortise the same weight reads. Paged attention — because contiguous KV allocation wasted 60–80% of the memory that determines how many sequences you can hold. Quantisation — because halving the weight bytes roughly doubles the decode ceiling. Prefix caching — because prefill you don't do is prefill you don't pay for.
Then I'd check the KV cache arithmetic, because it usually turns out to be the binding constraint rather than the weights. 2 × layers × KV_heads × head_dim × bytes. For a 70B-class model that's about 320 KiB per token. On an 80 GB card with FP8 weights at 70 GB, you have ~10 GB left, which is about 32,000 tokens of KV total across all concurrent requests. Eight users at 4K context and you're full. FP8 KV quantisation halves that and roughly doubles concurrency — usually the single highest-leverage setting.
Then, in order of effort: prompt caching, which is often free money if the prompt is ordered correctly; FP8 weights, which the largest published study found effectively lossless at all scales; batch API where latency permits, typically 50% off; and routing — most requests do not need the frontier model, and reasoning tokens bill at the output rate, which is 5–8× input.
And one thing I'd check before optimising: whether reasoning effort is set globally. Chain-of-thought inflates token cost 2–5× and adds seconds of latency with no measurable accuracy gain on most production tasks. Effort should be a routing decision per task class, not a default — and the threshold should come from a measured accuracy-vs-cost sweep on your own evals, because verbalisation overhead varies about ninefold across models at similar accuracy.
What is being tested
Whether you reason from the roofline rather than reciting techniques, and whether you know the KV cache — not the weights — usually caps concurrency.
Our prompt cache hit rate is near zero and nobody knows why.
What you said
A strong answer
Almost always prompt ordering, and it's usually one line of code.
Every vendor's implementation is a prefix cache. It matches from the start of the rendered prompt. So anything volatile near the front invalidates everything behind it, no matter how carefully you placed cache markers.
The usual culprits, in the order I'd look:
A timestamp or date in the system prompt. datetime.now() at position zero takes your hit rate to exactly zero.
An unsorted json.dumps() anywhere in the prefix — Python dict ordering is insertion-ordered, so the same logical content serialises to different bytes across runs.
A per-user or per-session identifier placed before the shared content instead of after it.
A tool list that varies by user. This one is particularly expensive because tool definitions are rendered at position zero, so changing them invalidates the system prompt and the entire message history too. Adding one tool to an agent can double the bill overnight.
Then the silent failures, which are the ones people never find:
Below the minimum prefix length there is no error at all — you just get no cache, and the usage field reports zero cached tokens. And the minimum is model-dependent and not monotonic across tiers: a newer larger model can have a lower minimum than an older smaller one. Porting a caching design between models without re-checking is an invisible regression.
Non-text settings can invalidate. Reasoning effort and output format are invalidators on some providers — change effort mid-session and you drop the cache even though the prompt bytes are identical.
And concurrent identical requests all miss, because the entry only becomes readable once the first response starts streaming. If you fan out fifty identical calls, you pay full price fifty times. Warm it with one request first.
What I'd actually do: treat the prefix as a build artefact — construct it deterministically, sort every serialised structure, put everything per-request at the end — and then assert on the cached-token count in an integration test. A cache-hit-rate regression is invisible in every other signal and shows up only on the invoice.
The economics justify the care: at a 1.25× write and 0.1× read, a five-minute cache pays for itself after a single reuse.
What is being tested
Whether you know it is a prefix cache and reason from that, plus the silent failure modes. The build-artefact framing and the test assertion are what separate someone who has operated this.
We changed a prompt and our eval went from 71% to 73%. Do you ship it?
What you said
A strong answer
Not on that evidence, and I'd want to know the eval size before saying anything else — because there's a good chance that number cannot resolve a two-point difference.
The arithmetic. For two proportions at 80% power and the usual significance level, unpaired, you need roughly 15.68 × p(1−p) / d² per arm. At p ≈ 0.5 and d = 2 points that's about 9,800 examples per arm. If this is a 500-item eval, a two-point move is comfortably inside the noise.
But I'd also fix the design, because unpaired is the wrong test here. Both versions ran on the same questions, so use a paired test — n ≈ 7.85 × discordance / d². If the two versions agree on 90% of items, detecting 2 points needs roughly 2,000 rather than 9,800. Pairing cancels per-item difficulty and is typically a 2.5× saving for the same power.
And the counter-intuitive corollary worth stating: sample size is driven by discordance, not accuracy. Two very similar systems need fewer examples to separate than two dissimilar ones at the same gap, because only the disagreements carry information.
Then the thing most people miss entirely: clustering. If several questions come from the same document or passage, the items aren't independent, and naive standard errors are wrong — correctly clustered errors can be more than three times larger. A great many reported eval improvements sit inside a properly computed error bar.
So what I'd actually do. Run the paired test and report the confidence interval, not the point estimate. If it straddles zero, say so plainly. Then decide whether to ship on other grounds — is it cheaper, faster, simpler? Those are real reasons, and 'it might be slightly better' is not.
And I'd put error bars on the dashboard permanently, because the underlying problem is cultural rather than statistical: a team that reads point estimates will chase noise indefinitely, and every chase costs a week.
What is being tested
Whether you know the power arithmetic, use a paired test, and are willing to say 'this is noise' about your own result.
Long context windows are huge now. Do we still need RAG?
What you said
A strong answer
Yes, and the reason is measurable rather than a matter of taste — but the honest answer is that the question is a false binary and the production architecture is neither.
Three independent lines of evidence that the advertised window is not the usable window.
Effective context length. A benchmark that defines it as the longest input where a model stays above a quality threshold found that although every model tested claims at least 32K, only about half hold up at 32K — and a model advertising 128K frequently has an effective length around half that.
Associative versus lexical retrieval. A benchmark that rewrites needle-in-a-haystack so the needle and question share minimal wording — forcing association rather than string matching — found most models below 50% accuracy at quite modest lengths. Models that ace vanilla needle-in-a-haystack fail this, which means high NIAH scores are close to meaningless as evidence of long-context competence. If someone cites NIAH at you, that's the response.
Context rot. Measured degradation with position and distractor density — one frontier model going from over 99% at short context to around 70% by 32K.
Then the cost side, which is the chapter-one arithmetic at scale: attention is quadratic in prefill, and a cached million-token session is on the order of a hundred gigabytes of KV state. Even with caching, you are paying to attend over a great deal of irrelevant material on every turn.
Where each genuinely wins. Long context wins when evidence is distributed across many documents and the task requires aggregation — no retrieval step can select the right subset when the answer depends on all of it. Retrieval wins when evidence is sparse and retrieval is accurate, and it is the only option at all for data that is fresh, private, or must be permission-filtered per user. That last one is often decisive and gets forgotten: you cannot put a document in the context window that this user is not allowed to see.
What I'd actually build in 2026: retrieve generously into a large window rather than retrieving precisely into a small one. Retrieval as an attention prior rather than a hard filter. That removes most of the brittleness of top-k selection — you're no longer betting everything on the chunk ranked third — while staying well inside the effective context length where the model still works.
What is being tested
Whether you know that effective context length is measurably shorter than advertised, and that NIAH scores do not support long-context claims.
Your agent stopped calling tools. It returns valid, confident answers. Nothing errors.
What you said
A strong answer
That specific symptom — valid output, no errors, tools silently unused — has a documented cause worth knowing by name: constraint priority inversion.
The mechanism. Tool calling isn't a special pathway. The model emits delimiter tokens it was fine-tuned to produce, followed by JSON, and the serving layer parses those tokens back into a structured call. Separately, strict structured output works by grammar-constrained decoding: a JSON schema is compiled into a finite-state machine, and at each decoding step any token that would violate the grammar has its logit set to negative infinity.
Put those together and the failure is inevitable: the schema mask makes the tool-call delimiter tokens unreachable. The model cannot emit a tool call because the grammar forbids the token that starts one. Published 2026 research found several open-weight models stopping tool use entirely while maintaining perfect schema compliance.
Why it's so hard to spot: nothing fails. The output is valid JSON, the schema is satisfied, latency looks fine. The agent has quietly become a chatbot answering from parametric knowledge — which is worse than an error, because the answers are plausible.
The fix is two-pass execution: an unconstrained tool phase, then a schema-constrained response phase. Never both at once.
How I'd diagnose it if I didn't already know: check whether tool calls appear in the raw token stream before the parser, not in the parsed output. If the delimiters never appear, it's decoding-level. If they appear and don't parse, it's the parser.
And the detection I'd add so it can't recur silently: alert on tool-call rate per session as a first-class metric. Any agent whose tool-use rate drops to zero is broken regardless of whether anything errored — and this class of bug, where the system produces confident valid output while doing none of the work, is the one that survives longest in production precisely because every conventional signal looks healthy.
What is being tested
Whether you understand tool calling as token emission plus parsing, which makes the interaction with constrained decoding obvious rather than mysterious.
How much do you trust published agent benchmark scores?
What you said
A strong answer
Considerably less than I did a year ago, and for a specific documented reason rather than general scepticism.
In 2026 a university group published work in which an automated scanning agent achieved near-perfect scores on eight major agent benchmarks without solving a single task. Not by being clever about the problems — by attacking the harness.
The methods were mundane and each is a design lesson. On a terminal benchmark, binary-wrapper trojans that intercepted dependency installation and fabricated test results — 100%. On two software-engineering benchmarks, an injected test-configuration file with hooks forcing every test to pass — 100%. On a web benchmark, file:// URLs reading gold answers straight out of the task configuration, plus prompt injection against the LLM judge. On a field-work benchmark, the validator only checked that the last message came from the assistant. On a general-assistant benchmark, public validation answers plus answer normalisation that stripped all whitespace and punctuation.
Corroborating evidence: an independent evaluation organisation found a frontier model reward-hacking in roughly 30% of runs — stack introspection, monkey-patching graders, operator overloading.
The framing that matters isn't 'models cheat'. It's that these benchmarks were built with the evaluation logic reachable from the agent's execution environment, and an agent optimising for a score will find that path — exactly as any optimiser finds the cheapest route to its objective. That's not misbehaviour, it's the objective being wrong.
So the reading rule when I see an agent score: ask what would happen if the agent attacked the harness instead of the task. If I can't answer, the number isn't evidence.
And the remediation list doubles as the spec for any internal agent eval, which is the practically useful part: complete isolation between the execution environment and the evaluation environment; reference answers never present in task config; no evaluation of untrusted strings; sanitised inputs to any model judge; read-only evaluation infrastructure; and adversarial testing of the harness before you trust a number from it.
I'd add one caveat against over-correcting: this doesn't mean agents don't work. It means the published numbers overstate them by an unknown amount, and the only scores I'd act on are ones from a harness I've adversarially tested myself.
What is being tested
Whether you know the result, and whether you can turn it into a design spec rather than into cynicism.
When would you fine-tune rather than use retrieval or prompting?
What you said
A strong answer
The rule that resolves most of these arguments: fine-tuning is for form, not facts.
Why, mechanically. Singular-value analysis shows that full fine-tuning's weight perturbations have ten to a hundred times higher rank than typical LoRA ranks. A low-rank adapter structurally lacks the capacity to absorb substantial new knowledge, and no amount of training data changes that. What it can do very well is shift behaviour, style, output structure and refusal patterns.
The measured comparison I'd cite, from a medical-reasoning study: base model 61.0% → retrieval 89.0% → fine-tuning plus retrieval 90.1%. Retrieval did essentially all the work; fine-tuning added about a point on top. Independent work agrees fine-tuning alone doesn't match retrieval for less-common knowledge.
So the sequence is prompt → retrieve → fine-tune → distil, and you should be able to say why the previous step failed before taking the next one.
Where fine-tuning genuinely wins:
Fixed output schemas where prompting is unreliable — a few thousand examples collapse variance that no prompt achieves.
Long stable system prompts — folding a 5,000-token preamble into weights removes it from every request. Though I'd check prompt caching first, since that's a config change rather than a training run.
Distilling a frontier model into a small open-weight model for one narrow task. This is the highest-return case in 2026 — an order of magnitude off inference cost at comparable task quality.
And the honest caveat on distillation: reported results have students retaining 70–85% of the teacher's broad-knowledge performance but only 50–70% of hard reasoning. Distillation transfers breadth much better than hard reasoning. So distil for throughput on well-defined tasks and route the difficult tail to the frontier model — routing between them usually beats choosing one.
One practical warning. LoRA learns less and forgets less than full fine-tuning, and forgetting is inversely related to rank. So evaluate on held-out general benchmarks, not just your target task — otherwise the degradation is invisible until a user finds it.
What is being tested
Form-not-facts, the mechanistic reason, and whether you evaluate for forgetting rather than only for the target task.
Copy-on-write or merge-on-read?
What you said
A strong answer
It's a straight trade between write amplification and read amplification, and the deciding question is usually organisational rather than technical.
The concrete case. Delete 1,000 rows scattered across 500 files of 256 MB each. Copy-on-write rewrites all 500 files — roughly 128 GB written to remove about 100 KB — and readers afterwards see clean files with zero merge cost. Merge-on-read writes a deletion vector marking those rows: kilobytes written, and every subsequent reader applies the vector.
What each format's default tells you. Iceberg defaults delete, update and merge to copy-on-write — a batch, read-heavy assumption. Delta on Databricks defaults new tables to deletion vectors — a frequent-DML assumption. Hudi gives you merge-on-read plus a separate read-optimized query type so you can opt out of merge cost per query. The defaults encode the workload each designer had in mind, which is a more useful signal than the feature matrix.
So: CoW when deletes and updates are rare, or touch whole partitions, or readers are latency-critical. MoR when a small number of rows scattered across many large files change frequently — the CDC-upsert and GDPR-delete shape, where CoW rewrites gigabytes to change kilobytes.
And the failure mode that decides it in practice. Merge-on-read accumulates deletes silently and read performance degrades monotonically until someone compacts. Nothing warns you; the table just gets slower over months.
So my actual deciding question is: does anyone own the compaction job? If there's no named owner and no monitoring on delete-file count or vector cardinality, I'd choose copy-on-write even where merge-on-read is technically better — because a format that requires maintenance nobody performs is worse than one that doesn't.
The related detail worth flagging in the same conversation: on Delta, partitioning disables row-level concurrency, which needs deletion vectors and an unpartitioned table. So a team that partitions for pruning may have quietly traded away concurrent writes without noticing.
What is being tested
Whether you frame it as a maintenance-ownership question rather than only a performance one.
Our Snowflake bill doubled and query volume didn't change.
What you said
A strong answer
Almost never inefficient queries. In every review I've seen, it's configuration — and there are about six candidates.
Auto-suspend, in both directions. Too long and you pay for idle warehouses at full size, multiplied by cluster count on a multi-cluster warehouse. Too short and you drop the warehouse's local SSD cache constantly, so every query rescans from remote storage — slower and more expensive for identical work. These two failure modes point in opposite directions, which is why 'set auto-suspend as low as possible' is confidently wrong. Around ten minutes is the documented compromise for BI.
Serverless features stacking. Classic automatic clustering, materialized-view maintenance and search optimization each bill at a 2× multiplier. On the same hot table that's six times serverless compute running alongside your warehouses. Each was individually justified; nobody added them up. I'd query the serverless usage views by feature and by object first — this is usually where the money is.
Ingest billing on compressed text. Text files are charged at uncompressed size, binary formats at observed size. So gzipped JSON is billed as though you never compressed it. If someone changed the landing format, that alone can move the number.
Cloud services crossing the 10% threshold. Free up to 10% of daily warehouse credits, charged beyond. Usually caused by very high volume of tiny metadata queries — a BI tool polling the information schema on a short interval.
A warehouse generation change. Newer-generation warehouses carry a multiplier — around 1.35× on some clouds. Someone 'upgrading' warehouses gets an immediate 35% increase unless queries actually got more than 35% faster. That's a checkable claim per workload, not a given.
And the newest one: interactive warehouses have a 60-minute minimum on start or resume, against one minute for a standard warehouse. Treat one as an ordinary warehouse and start it a few times a day and you've bought hours you didn't use.
How I'd actually find it: the account usage views broken down by warehouse, by serverless feature and by day, comparing the two weeks either side of the change. The shape of the increase — step change versus gradual — tells you whether it was a setting or a growth.
What is being tested
Whether you go to configuration rather than query tuning, and whether you know auto-suspend cuts both ways.
Our dashboards are slow. Where do you start?
What you said
A strong answer
By finding out which of four layers is slow, because the four fixes are disjoint and three of them will do nothing.
1. The source query. One visual is slow, and the query is slow when you run it directly against the warehouse. Fix it there — pruning, clustering, pre-aggregation. Nothing in the BI tool helps.
2. The BI engine. Queries are fast at the source and slow in the tool. That means row-by-row evaluation, huge intermediate results, or an oversized model. On Power BI specifically, capture the generated storage-engine queries: work done in the storage engine is nearly free, and rows materialised for the single-threaded formula engine are where slowness lives.
3. Rendering. Queries fast, screen slow. This is the one people miss entirely — mark count. A scatter plot with 400,000 points is 400,000 objects to lay out and draw, and no amount of database tuning touches it. It is also a design problem, because nobody can read 400,000 points; binning makes it faster and more legible.
4. Concurrency. Fast alone, slow at 9am. Every user issuing live queries at once.
Then the fixes that beat tuning, in order of return:
Pre-aggregate to the grain the dashboard actually shows. A dashboard of monthly revenue by region does not need a row-level fact table. This single move solves more performance problems than everything else combined, and it gets skipped because it feels like cheating.
Default the date filter to a narrow range. A dashboard loading three years because the filter defaults to 'all' is doing enormous work for someone who wanted this month.
Cut the visual count. Every worksheet is at least one query. Fourteen visuals and six filters is a lot of work per page view, and splitting it into two focused dashboards is usually better design anyway.
And the thing I'd check that nobody checks: whether the aggregation is actually being hit. A single mismatched data type or unexpected grain makes the engine fall through to the detail table silently, and you have paid for the aggregate and get none of the benefit.
What is being tested
Whether you diagnose the layer before optimising, and whether you know rendering and mark count are a separate problem from query speed.
Finance asks for real-time dashboards. How do you respond?
What you said
A strong answer
With one question before any design work: what decision do you make differently if this number is an hour old? In a finance context the answer is almost always none.
Why it matters so much. Live query makes every user interaction a warehouse query. A dashboard with eight visuals and four slicers generates dozens of queries per page view, per user. At forty finance users clicking filters, the warehouse bill rises materially and the dashboard becomes slower than an imported model would have been. You have paid more for a worse experience to satisfy a requirement nobody examined.
What 'real-time' usually means when you ask: not yesterday's. An hourly refresh satisfies it completely.
The middle answer that satisfies almost everyone: an imported model refreshing hourly, with a visible 'data as of' timestamp on the dashboard. Most of the anxiety behind the request is not knowing how stale the number is — and a small label solves that for nearly nothing. It's also the cheapest data-quality control there is, because it makes a failed refresh visible to the person best placed to notice.
Where live query is genuinely right: operational monitoring — a fraud queue, a fulfilment exception board, a live incident view. Those are real requirements, and they're usually better served by a purpose-built operational display than a BI dashboard, because the latency and concurrency profile is different.
The order I'd ask the questions in: how fresh does this genuinely need to be, then how many concurrent users, then how large is the model. Only then choose a storage mode. Choosing the mode first and discovering the concurrency later is how the expensive version happens.
And I'd say the pushback matters more than the technical answer here. Agreeing to 'real-time' because a stakeholder asked for it is how a team ends up defending a slow, expensive dashboard for two years.
What is being tested
Whether you interrogate the requirement rather than satisfying it, and whether you know live query cost scales with interactions rather than data volume.
We have three BI tools and three different revenue numbers. What do you do?
What you said
A strong answer
Fix the definitions first. Consolidate tools second, if at all — and I'd argue that sequencing hard, because it's usually reversed.
Why consolidation-first fails. The three tools are a symptom, not the disease. If you migrate everyone onto one platform and each team rebuilds its own model with its own definition of revenue, you have spent a year and a large amount of money and still have three numbers. This happens often enough to be predictable.
The actual problem is that nobody owns the definition. One report excludes internal transfers and another doesn't, and both authors were correct according to their own understanding. That's not a data quality problem or a tool problem — it's a definition-ownership problem.
What fixing it looks like. Put the definitions somewhere every tool can reach — the warehouse, or a semantic layer. The technical reason a semantic layer beats a set of views is grain-correct aggregation: conversion rate is orders over sessions, and you cannot pre-aggregate that by day and roll it up to a week, because the average of seven daily ratios is not the weekly ratio. Same for distinct counts and anything windowed. The layer stores the components and computes at the requested grain.
And the sequencing insight: shared definitions make consolidation optional. Once every tool queries the same certified metrics, three tools is a licensing inefficiency rather than a correctness problem — a much smaller and more patient thing to solve.
The failure mode to plan for is social, not technical. An analyst needs a number by Thursday. Going through the semantic layer means learning the modelling language or waiting for someone who knows it; writing the SQL takes twenty minutes. They write the SQL, and a new definition enters the world. The governed path has to be faster than the ungoverned one, or governance loses every time. So: cover the top twenty metrics extremely well rather than every metric adequately, make adding one take hours not a sprint, and measure the proportion of queries going through the layer — that number, not the count of metrics defined, tells you whether it worked.
I'd also flag the politics honestly. Consolidation is popular with executives because it looks decisive and has a number attached. Definition work is unglamorous and has no launch date.
What is being tested
Whether you separate the symptom from the cause, know why a semantic layer beats views technically, and anticipate that the failure mode is people routing around it.
How would you deploy a change to a service safely?
What you said
A strong answer
I'd pick the strategy from the failure I'm actually worried about, and then deal with the thing that makes all of them harder than they look.
The strategies, and what each is for. Rolling — replace instances in batches, no extra infrastructure, and the default for an ordinary service. Blue-green — two full environments and a router flip, which buys you instant certain rollback at the cost of double infrastructure. Canary — 1% then 5% then 25%, real traffic with a small blast radius, and the right answer when you have enough traffic to produce a signal. Feature flags — deploy dark and release to a cohort, which separates deploying code from releasing behaviour entirely.
Now the part that matters more than the choice. Every one of those except a full stop-and-replace runs two versions of your code simultaneously — during the rolling window, throughout the canary, across the blue-green drain. So for that entire period: database schema changes must be backward compatible, API changes must be additive, message formats must be readable by both versions, and cached data must be compatible with both.
Which is why schema changes go through expand-and-contract, in three deploys. Add the column nullable and dual-write while still reading the old. Backfill, then read from the new while still writing both. Then, days or weeks later, stop writing the old and drop it. And I'd flag that a rename is a drop and an add — there is a moment when running old code sees neither, which is why renames break things that 'should have been safe'.
The asymmetry to state explicitly: stateless services roll back, data does not. You can redeploy yesterday's binary in a minute; you cannot un-write the rows it produced. Any change that mutates data needs a written reversal plan before the change, because writing one during an incident is how bad decisions get made.
On canary specifically, if I'm asked: compare against a baseline running the old version at the same moment, not against last hour's metrics — otherwise you've confounded your change with time of day and load. Gate on error rate, tail latency, saturation and at least one business metric, because a deploy that halves conversion while every technical metric stays green is exactly what a technical-only gate misses. And match the bake time to the slowest failure you fear: five minutes catches a crash loop and will never catch a memory leak.
What is being tested
Whether you name version skew unprompted. Most candidates list the strategies; few mention that two versions run concurrently and what that implies for schema and API changes.
Our CI pipeline takes 45 minutes. Fix it.
What you said
A strong answer
I'd measure before changing anything, because the time is almost never where people assume — and then I'd argue about which number we're actually trying to improve.
First, instrument per stage and look at the distribution, not the mean. A p50 of 45 minutes and a p95 of 45 minutes are different problems.
Then work cheapest-to-try. Caching is almost always the biggest single win — most pipelines reinstall dependencies from scratch every run, and a warm dependency cache plus a content-hashed build cache routinely removes half the duration for a day's work. Parallelism next: lint, typecheck and unit tests almost never depend on each other and are frequently run in sequence for no reason. Test selection is where the largest wins are — affected tests on every push, full suite on merge to main — and it needs a dependency graph, which is why build systems that compute one exist. Bigger runners is the least elegant and often the most cost-effective: an engineer waiting fifteen minutes costs more than a larger machine.
And measure queueing separately, because time waiting for a runner doesn't appear in the build log and is invisible on every dashboard. It can be a large share of the total.
Then the reframe I'd push on. Pipeline duration is the number the platform reports; it is not the number a developer experiences. Time from push to merged includes queueing, flaky reruns, waiting for a reviewer and the merge queue — and in most organisations it's dominated by the parts that are not the build. A team can spend a quarter optimising build speed and move the metric that matters by almost nothing.
I'd also look at flakiness, because it's usually entangled with this. A flaky test does something worse than fail — it teaches people that red might mean nothing, and once re-running is habitual, a genuine regression gets re-run away with the rest. Quarantine flaky tests out of the blocking suite immediately with a ticket and an owner, and track the flake rate as a visible number. Silently retrying at every layer means you no longer know what the suite is telling you.
And set a budget with an alert, because pipelines regress silently as people add steps. Without one you'll be back here in six months with a pipeline nobody deliberately made slower.
What is being tested
Whether you measure first, know that caching and test selection dominate, and reframe onto push-to-merged rather than build duration.
Where would you put security checks in the pipeline?
What you said
A strong answer
At the earliest point where each check can run reliably — which is a narrower claim than 'shift left', and the difference is what makes it work.
Where each belongs. Secret scanning — pre-commit hook and in CI, because hooks are trivially bypassed and you need both. Dependency scanning — on the pull request, since it needs the resolved lockfile. Static analysis — on the pull request, scoped to the diff. Infrastructure policy — against the plan, before apply, because the code alone doesn't tell you what will happen to production. Container scanning — at build and again on a schedule, since a vulnerability disclosed after you built is invisible to a build-time-only scan. Dynamic analysis — against something deployed.
Now the rule that decides whether any of it survives. A check that blocks a build must have a very low false-positive rate. A check with a high false-positive rate must not block. Get that backwards and you get the standard outcome — developers acquire a habit of adding suppression comments, or security grants a blanket exception, and the control becomes decorative while still costing everyone time. A control that is routed around is worse than no control, because it also creates false assurance.
So: scope static analysis to the diff rather than the whole repository — full-repo scanning on every PR produces a wall of pre-existing findings nobody reads. Block only on high-confidence, high-severity findings. Route everything else to a backlog with an owner. And never turn on blocking for a new scanner immediately — run it report-only long enough to measure its false-positive rate on your codebase, then decide what blocks.
The link I'd flag as most underestimated is the build itself. Your CI executes arbitrary code from every dependency you install — install scripts, build plugins, code generators — with whatever credentials the job holds. A runner with a long-lived cloud credential and a deploy token is an extremely attractive target reachable by anyone who can get a package into your dependency tree. Ephemeral runners, no secrets available to fork builds, short-lived federated credentials rather than stored keys, and least privilege on the job.
And the one that's trivial to fix and usually isn't: dependency confusion. If your internal package name isn't reserved on the public registry, an attacker publishes it there with a higher version and your resolver may prefer it. Reserve the namespaces publicly, forbid public fallback for internal scopes. Both one-time changes.
What is being tested
The false-positive rule. Candidates who say 'shift left, scan everything' have not watched a noisy gate get suppressed into irrelevance.
Walk me through how infrastructure as code actually works, and where it goes wrong.
What you said
A strong answer
The loop is: your config says what you want, the state file says what the tool thinks exists, a refresh reads what actually exists, the plan is the diff of all three, and apply is the only mutating step.
The state file is the design decision everything follows from. It maps resources in your configuration to real objects by their provider-assigned identifiers. That mapping isn't derivable — names aren't unique and tags can be edited — so without it the tool has no way to know that web_server in your code is that particular instance in the account.
Which is why all three failure modes are really about state.
Drift. Someone changes something in the console during an incident, reasonably, and the next plan proposes to undo their fix. Detect drift on a schedule rather than discovering it at apply time, make console access read-only for managed resources, and have a documented break-glass path — because forbidding emergency changes just means they happen and nobody tells you.
State loss. Lose the file and the tool believes nothing exists and will happily create a second copy of production. Remote, versioned, encrypted, with locking so two people can't apply at once. Import can rebuild it, slowly and manually, so the backup matters more than the recovery procedure.
Secrets in state. This one surprises people every time — the state file records resource attributes including generated passwords and connection strings, in clear text. Treat it as a secret store: encrypt at rest, restrict reads as tightly as a vault. Better still, have the infrastructure create a secret reference and let the value be generated and held elsewhere.
The practice that separates teams who trust this from teams frightened of it: review the plan, not just the code. Code expresses intent; the plan expresses what will happen to production, and they differ more often than expected — drift, a module version that moved, a provider default that changed. Post the plan onto the pull request. And search it for 'forces replacement': many changes are updates in place, some force destroy-and-recreate, and for a database that means deleting the real thing.
On layout — split state by blast radius and change cadence. Networking changes rarely and is catastrophic to break; application infrastructure changes daily and is cheap to fix. They don't belong in one apply. Reference across boundaries with a data lookup rather than shared state. But the counterweight is real: too many small states and one logical change needs six coordinated applies in the right order, which is its own fragility. It's a judgement about your organisation, not a rule.
And policy-as-code on the plan is what turns infrastructure standards from documentation into something that cannot be violated — a much better answer to 'how do you enforce cloud standards' than a review checklist, because it runs on every change without a human being available.
What is being tested
Whether you understand why the state file must exist, and whether you review plans rather than code. The secrets-in-state point is a reliable experience signal.
How do you know whether your internal platform is working?
What you said
A strong answer
One question, and it is uncomfortable: if the platform became optional tomorrow, would teams still use it?
If yes, you built a product. If no, you built a tax — and the adoption number you're reporting upward measures compliance, not value.
The distinction underneath it is paved road versus gate. A paved road makes the standard path the fastest path; a team takes it because it's quicker, and can leave when their needs genuinely differ. A gate makes it the only permitted path; a team whose needs fall outside raises a ticket, waits six weeks, and eventually routes around — and now you have shadow infrastructure you can't see, which is worse than the non-standard thing you were preventing.
The two look identical on an architecture diagram. The difference shows up entirely in what happens to the team at the edge of the standard case.
What I'd measure instead of teams onboarded, which measures the platform team's activity rather than anyone's outcome:
Time to first deploy for a brand-new service — and measure it by actually doing it, quarterly, with a stopwatch, rather than believing the documentation. If it takes three days, the platform is a documentation site.
Time to first commit for a new engineer — captures documentation, environment setup, pipeline and review culture in one number.
Self-service rate — the proportion of requests completed with no ticket. The clearest measure of whether you built a product or a service desk.
Voluntary retention — teams that could leave and didn't. The honest version of adoption.
And one qualitative practice worth more than any of them: a quarterly survey asking developers what wastes their time, published, with the top item actually fixed. Platform teams that do this outperform ones building from a roadmap, for the obvious reason — they're solving problems people have rather than problems they imagined.
The failure patterns I'd watch for in myself: abstracting before three teams have solved the problem their own way, so the abstraction is a guess. Owning incidents without owning the code, which makes you an operations team with a different name. And having no product management — a platform is a product with internal customers who can in practice say no, and if nobody prioritises, it becomes whatever the loudest team asked for last.
What is being tested
Whether you can ask the optional-tomorrow question about your own work, and whether you measure outcomes rather than the platform team's output.
How do you choose between functions, containers and Kubernetes?
What you said
A strong answer
By the ceilings first, then the duty cycle, and I'd lead with the architectural difference that matters more than either.
The ceilings, because they eliminate options before anything else. Maximum runtime — 15 minutes on the classic serverless function model, around 60 for serverless containers, unbounded above that. Request and response size — one major provider caps a synchronous function response at 6 MB, which rules functions out of a great deal of media work regardless of anything else. And on one cloud there's a hard ~230-second ceiling on an HTTP-triggered function response that comes from the load balancer in front of it, so raising the function timeout does nothing. Discovering that after building a long-running synchronous endpoint is expensive.
Then the thing most candidates never mention. A function handles one request per execution environment; a container handles many. Three consequences: per-request memory is instance memory, so occasional large requests set the size you pay for on every request. A connection pool cannot pool — 200 concurrent requests means 200 database connections, which is why managed connection proxies exist and how serverless applications take down their own databases. And I/O-bound waiting is billed idle time, where a container serving 200 concurrent requests amortises it.
Then duty cycle for cost. Below roughly 10% busy, serverless wins clearly. Above roughly 50%, committed instances win clearly. Between them it's cold-start tolerance and operational cost — a team of five that would otherwise be patching operating systems has a real reason to pay the premium. 'Serverless is cheaper' and 'serverless is more expensive' are both true and the duty cycle is what resolves them.
And I'd flag Kubernetes honestly: it's the wrong answer for a small team with a handful of services. The operational surface is real and it doesn't shrink because the control plane is managed.
One cheap lever I'd mention: ARM-based compute is materially cheaper on the same platform — exactly 20% less on both CPU and memory on one provider's serverless containers. For anything that compiles cleanly, that's a one-line change for a fifth off.
What is being tested
Whether you know the one-request-per-environment difference and reason from ceilings and duty cycle rather than preference.
Pods are stuck Pending and the nodes have plenty of CPU and memory.
What you said
A strong answer
Almost always IP address exhaustion, and the mechanism is worth knowing precisely because the symptom points nowhere near the cause.
On a flat networking model — where every pod gets a real routable address from the same network as your other resources — the maximum pods per node is a property of the instance type, not of Kubernetes. The formula is roughly (network interfaces × (addresses per interface − 1)) + 2. A mid-size instance with three interfaces at ten addresses each tops out around 29 pods, with most of its CPU and memory unused. The scheduler has nowhere to put the pod and says so in the events, but the message is about IPs and everyone is looking at resource metrics.
First thing I'd do: kubectl describe pod and read the events. They name the reason explicitly. Then check allocatable pods per node against the instance type's interface limits.
The fix is prefix delegation — hand each interface slot a block of sixteen addresses instead of one, multiplying density roughly sixteenfold. It's also faster, because it makes far fewer control-plane API calls when starting pods. Two cautions: it's close to a one-way door, since you can't downgrade the networking plugin afterwards without replacing every node; and mixing prefix-enabled and non-prefix nodes advertises inconsistent capacity, so the clean path is new node groups rather than rolling existing ones.
The arithmetic I'd want done before the next cluster: 500 nodes at 110 pods each is roughly 55,000 addresses. That's a very large block for one cluster's pods, and it's why flat-model clusters routinely need secondary ranges bolted on later. Subnet sizing is a design-time decision that is painful to reverse — an hour spent on it before the first cluster beats a quarter spent on it afterwards.
The alternative is an overlay model, where pods sit on a separate range and translate on the way out. Address space stops being a constraint entirely; the trade is that pods aren't directly reachable from peered networks, which is disqualifying if an on-premises system must call into a pod.
Other causes worth ruling out if the events say otherwise: an unsatisfied node selector or affinity rule, a taint with no matching toleration, a volume bound to a different availability zone, or a resource quota on the namespace.
What is being tested
Whether you go to the pod events first, and whether you know that max-pods is an instance-type property on flat networking.
How would you secure the software supply chain?
What you said
A strong answer
I'd work the chain link by link, and I'd start with the one people underestimate.
The link that is underestimated is the build itself. Your CI executes arbitrary code from every dependency you install — install scripts, build plugins, code generators — with whatever credentials the job holds. A runner with a long-lived cloud credential and a deploy token is an extremely attractive target, reachable by anyone who can get a package into your dependency tree. So: ephemeral runners, no secrets available to builds from forks, short-lived federated credentials rather than stored keys — which removes the class rather than protecting it — and least privilege on the job, since a test job does not need deployment permissions.
Dependencies. Lockfiles with integrity hashes, pinned versions, an SBOM, and reviewing what a dependency actually does before adding it. And dependency confusion, which is trivial to fix and usually isn't: reserve your internal namespaces on the public registry and forbid the resolver from falling back to public for internal scopes. Both are one-time changes.
Artifacts. Deploy by digest, not by tag. That is the portable control — a digest is immutable by construction and needs no registry feature, which matters because 'immutable tags' means four incompatible things across the major registries and two of the biggest enterprise binary stores don't enforce it natively at all. Sign artefacts and — this is the part people skip — verify at admission. Signing without verification is ceremony.
On provenance levels, be precise, because this is where confident wrong answers happen. The current specification has three build levels — there is no level four; that was a superseded draft. Hosted CI attestations give you level 2 out of the box, and level 3 requires reusable workflows for build isolation. 'We use hosted CI therefore level 3' is false and people say it in good faith.
And the question the whole programme exists to answer: 'a critical vulnerability was just announced in library X — are we affected, and where?' If you generate SBOMs and discard them, you've satisfied an auditor and helped nobody. Store them, index them, and rehearse the query — most teams discover the answer is 'we can't', and that discovery is worth more than the tooling purchase it prompts.
On regulation, the picture reversed and most material hasn't caught up. European obligations become binding in September 2026 for reporting actively exploited vulnerabilities, with the full regime including SBOMs in December 2027. The American attestation mandate was rescinded in early 2026. So if you're building to a compliance requirement, build to the European one — and note the dependency: you need SBOM capability about fifteen months before the deadline that names it, because you can't answer 'are we affected' within hours without one.
What is being tested
Whether you name the build step as the weak link, know that level 3 needs reusable workflows, and can state the regulatory position accurately.
Does platform engineering actually improve delivery?
What you said
A strong answer
The honest answer is that we don't know, and I think being able to say that precisely is more useful than either the vendor pitch or the cynical dismissal.
What the evidence actually is. A 2026 peer-reviewed literature review surveyed 88 sources across six years. Two findings matter. First, only about 2% were peer-reviewed work with platform engineering as its primary focus — academic research lags practice by two to three years, and essentially all the authoritative data comes from vendors and foundations. Second, and more pointedly, there is no peer-reviewed empirical evidence of effectiveness for scorecards — the governance mechanism at the centre of every commercial internal developer portal — despite universal adoption.
Large surveys do consistently show that mature platform practice correlates with better delivery performance. But longitudinal and interventional evidence is absent: there are no before-and-after studies tracking organisations through adoption. So the correlation cannot distinguish 'platforms cause performance' from 'high performers adopt platforms'. And the hypothesised productivity dip before the gain has never been quantified.
I'd also flag the specific numbers people quote. The widely-repeated market-share figure for the dominant open-source portal comes from a vendor survey of about a hundred engineering leaders. The '60–70% of portal initiatives fail' figures trace to marketing content with no primary attribution. If someone cites either at me, that's what I'd say.
So what would I actually claim? Two things a portal reliably delivers: discoverability — answering 'who owns this and where does it run' during an incident — and paved-path provisioning. Both are real and both justify the investment in most organisations above a certain size. What I would not claim is a delivery-metric uplift, because that chart has no independent support and the reviewers looked for it.
And the one consistent finding across the survey literature is that organisations treating the platform as a user-centric product rather than mandated infrastructure report better outcomes. Which lines up with the test I'd actually apply: if the platform became optional tomorrow, would teams still use it? If not, the adoption number measures compliance rather than value.
I'd rather build something engineers choose and measure discoverability and provisioning time honestly, than claim a productivity number I can't defend.
What is being tested
Whether you can distinguish correlation from causation in your own field's literature, and hold a position that is neither vendor enthusiasm nor cynicism.
Has AI made our engineering organisation faster?
What you said
A strong answer
The research says: faster at shipping, and less stable — and the mechanism matters more than the finding.
What reversed. For a decade the defining result in delivery research was that throughput and stability rise together — the speed-versus-safety tradeoff was false, because the same practices produced both. It was the single most quoted finding in the field. The 2025 research programme, across roughly five thousand professionals, found AI adoption has a positive relationship with throughput and a negative one with stability. The pairing broke.
The mechanism is the useful part. It's not that generated code is bad. It's that removing the coding bottleneck saturates every bottleneck downstream of it — review, testing, QA, release. If review capacity was your real constraint, AI makes your system worse. The researchers' framing is that AI is an amplifier: it magnifies existing strengths and existing weaknesses.
Which changes the question from 'should we adopt AI' to 'what is our next bottleneck, and have we widened it first?'
Why two teams report opposite experiences. Follow-up modelling puts gains at 35–40% on simple greenfield work and 10% or less on complex legacy code. That spread is the most useful number in the literature — the task mix explains most of the disagreement, and a team maintaining a fifteen-year-old system is not lying when they say it hasn't helped much.
And expect a dip before the gain — workflow adaptation plus the cost of reviewing generated output. Leaders should plan for it rather than read it as failure.
What I'd actually measure. Not lines or acceptance rate — those are activity, and Goodhart applies immediately. I'd watch change failure rate and time to restore alongside throughput, because the finding predicts exactly that pairing will diverge, and if it does I know where to invest. I'd also watch review latency and queue depth specifically, since that's the bottleneck the research says gets saturated.
One honesty note: over 80% of respondents believe AI increased their productivity. That's belief, self-reported. And 30% report little or no trust in AI-generated code. Both are findings about people rather than about output, and I'd be careful not to present the first as a measurement.
What is being tested
Whether you know the throughput/stability reversal and reason from the bottleneck mechanism rather than from a productivity number.
Walk me through threat modelling a new feature.
What you said
A strong answer
Four questions, forty minutes, and the output is a ticket rather than a document.
1 · What are we building? Draw the data flow on a whiteboard, badly, in ten minutes. Components, the data moving between them, where it is stored. Then — and this is the part that does the work — draw the trust boundaries: the points where the level of trust in the data or the caller changes. Internet to edge. Edge to service. Service to data. Tenant to tenant.
Almost every real vulnerability lives at a boundary crossing, because that is where an assumption about what to trust gets made. Inside a component the code trusts itself; across a boundary it has to decide, and the bugs are in that decision.
2 · What can go wrong? Six questions at each crossing — spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege. It's a checklist rather than an insight, and that's precisely the value: it catches the category nobody in the room was thinking about.
3 · What do we do about it? Mitigate, transfer, accept, or eliminate. Accept is a legitimate answer and writing it down is the point — an accepted risk with a name on it is a decision; an unexamined one is a surprise.
4 · Did we do a good job? Which mostly means: did the design change?
How I'd run it. Two engineers who built it plus one person who didn't — the outsider asks what the builders have stopped seeing. Whiteboard, not a diagramming tool, because a tool turns it into a drawing exercise. Forty minutes, timeboxed, accepting it won't be exhaustive. And the output goes into the feature's own ticket as three or four security requirements — not a separate wiki page, because work that lives with the ticket gets done and work that lives in a security repo does not.
When to trigger it: when a trust boundary changes — a new integration, tenant model, external caller, or class of data. That's a better trigger than a sprint cadence, which produces attendance rather than analysis.
And if it's an AI feature, two boundaries move. Retrieved content entering the prompt is untrusted input that will be read as instructions, and there's no parameterised channel to separate them — so it can't be fixed at the input layer, only by constraining what the model may do. And the model's output crossing into any tool is untrusted input to that tool.
What is being tested
Whether you lead with trust boundaries and end with a ticket. Candidates who describe producing a document have watched it happen rather than run it.
How would you secure a Kubernetes cluster?
What you said
A strong answer
I'd start by correcting the mental model, because it determines everything else: a container is not a virtual machine. It's a process on the shared host kernel, constrained by namespaces (what it sees), cgroups (what it consumes) and capabilities and seccomp (what it can do). The kernel is shared — so a kernel vulnerability is a container escape, and genuinely hostile multi-tenancy needs hardware virtualisation underneath rather than namespaces alone.
Then the four settings that remove most practical risk, which are a short block of YAML and are missing not because anyone disagrees but because the defaults are permissive:
Non-root user — root in the container plus any escape primitive is root on the node, and this is the single highest-value change. Read-only root filesystem — an attacker who can't write can't persist or drop tooling. Drop all capabilities and add back nothing; almost no application needs one, and if something breaks you learn immediately which. And no privilege escalation.
Then the RBAC audit, because cluster permissions look granular and several are administrator in disguise. Creating pods means running arbitrary code on a node with the node's identity — and 'developers can deploy' usually grants it. Reading secrets in a namespace means reading every credential in it. Creating a service-account token is a lateral-movement primitive. Modifying webhook configurations lets you disable every admission policy including the ones enforcing the settings above. And people restrict pod creation while leaving deployments and cron jobs open, which is the same permission one step removed.
So the practical audit is: who can create pods, read secrets, and modify webhooks? In most clusters that's far more people than anyone expects, usually through a group binding that made sense when there were three services.
Then workload identity — and this is usually my highest-value finding. If the cluster gives the node a cloud identity, every pod on that node inherits it, so a compromised sidecar has the permissions of the most privileged workload sharing its node. Federating each pod's own service account to the cloud provider fixes it and removes static credentials entirely. Related: can pods reach the cloud metadata endpoint? If so, any request-forgery bug in any workload becomes credential theft, and blocking it at the network layer is one policy.
Then admission control to make all of the above structural rather than remembered — starting in audit mode, because enforcing without seeing what you'd have rejected breaks deployments and burns the programme's credibility. With a documented, time-limited exception path, because some workload will genuinely need a capability.
Runtime detection last, deliberately. A cluster with the preventive controls enforced generates far fewer runtime alerts, and the ones it does generate mean something.
What is being tested
Whether you know a container shares the kernel, which RBAC verbs are secretly total, and that node-level cloud identity collapses your whole authorisation model.
Leadership says the team needs to move faster. What do you do?
What you said
A strong answer
I'd want to establish what 'faster' means before agreeing to anything, because the request usually has a specific frustration behind it — a missed date, a competitor shipping, a stakeholder who can't see progress — and those need different responses.
But the arithmetic almost always says the same thing. Lead time equals work in progress divided by throughput. If a team completes five items a week and forty are in flight, the average item takes eight weeks. That's arithmetic, not effort. Working harder doesn't change it — finishing more or starting less does.
So 'we need to move faster' almost never means people are working too slowly. It means too much is started and not enough is finished. I'd pull the cumulative flow and look at the band: if it's widening, lead time is rising; if a stage is bulging, that stage is the bottleneck — and it is very often code review rather than development.
The highest-return intervention is limiting work in progress, which shortens lead time without anyone working faster, costs nothing, and feels deeply wrong to everyone because it looks like doing less. The resistance is worth naming honestly: starting something feels like progress and is visible immediately; finishing takes longer and is only visible at the end. Individuals and managers both optimise for the visible signal, which is exactly how forty things end up in flight.
Then I'd check for the structural cause. If the team is blocked on other teams constantly, that's a dependency problem and no process change fixes it — the response hierarchy is remove it, break it with a stub or flag, invert it so the dependent team builds it, and only last schedule it. And if a programme has many scheduled cross-team dependencies, the team boundaries are wrong, which is an org-design conversation rather than a delivery one.
What I'd push back on. Adding people to a late project. Cutting testing. Estimating harder — velocity used as a target inflates immediately and permanently, because the number is defined by the people it measures. And I'd be careful with 'faster' if the team is already at a high change-failure rate, because throughput and stability can diverge and shipping more broken things is not faster.
And I'd give them something visible. Frequently the underlying complaint is not speed but opacity — a stakeholder who can't see progress assumes there isn't any. A forecast expressed as a confidence range from actual cycle-time history often resolves the conversation without changing anything about the work.
What is being tested
Whether you reach for the flow arithmetic rather than for motivation, and whether you check whether the real complaint is speed or visibility.
How would you make the case for a six-month platform investment?
What you said
A strong answer
Four elements, and the third is the one people skip and the one that usually wins it.
1 · The problem in the reader's units. Not 'our pipeline is slow' — 'engineers wait an average of 40 minutes per change; across 30 engineers and 400 changes a month that's roughly 260 hours, about 1.5 full-time engineers of pure waiting.' Convert to the unit the person holding the budget controls. And note that the real cost of a slow pipeline is the context switch, not the wait — a developer who learns of a mistake in forty seconds fixes it in context; one who learns after twenty-five minutes has started something else.
2 · The intervention and its honest cost, including people-time. A project that's 'free because we'll do it ourselves' costs whatever those people would otherwise have built. Pretending otherwise damages your credibility the second time you ask.
3 · What happens if we do nothing. Almost always omitted and frequently the strongest part. Costs that grow, risk that accumulates, a dependency going out of support, attrition. A case that only shows upside competes with every other upside; one that shows a rising cost of inaction has a clock on it.
4 · A confidence range with a named uncertainty. 'Seven to eleven months to recover the cost; the main uncertainty is how much of the estate needs migrating, and we'll know within three weeks.' More credible than a point estimate, and it protects you when the number moves.
The framing that consistently outperforms: offer options rather than a request. Do nothing, do the minimum, do it properly — each with a cost and a consequence. Decision-makers respond far better to a choice than to an ask, and three options makes you the person helping them decide rather than the person wanting something.
What I'd be careful about claiming. If it's a platform or developer-experience investment, I would not promise a delivery-metric uplift, because the honest state of the evidence is that platform maturity correlates with performance and has never been shown to cause it — there are no longitudinal or interventional studies. I'd claim the things that are demonstrable: time to first deploy for a new service, time to first commit for a new engineer, and the proportion of requests completed without a ticket. Measure those before and after with a stopwatch.
And I'd put a kill criterion in the proposal. 'If by month three we haven't moved time-to-first-deploy below X, we stop.' Volunteering that makes the rest of the case far more believable, and it's the thing that distinguishes someone asking for six months from someone who has run a six-month project before.
What is being tested
Whether you include the cost of inaction, offer options, and are honest about which benefits you can actually evidence.
Which frontend framework should we use?
What you said
A strong answer
I'd resist answering that until I know three things, because the framework is usually the least consequential decision in the room.
What kind of thing are we building? A content site that has to be indexed, a logged-in application where the first load happens once a day, or an internal tool that is mostly forms and tables. Those have genuinely different right answers — an islands-based content framework, a full application framework, and in the third case possibly no framework at all. The internal-tools case is the one teams get wrong most: a hypermedia approach appears on about a tenth of a percent of sites and is architecturally correct for a surprising number of admin panels.
Who is the team? This dominates. Hiring depth is a real engineering constraint, and it is the honest argument for React — not that it is technically superior, but that you can staff it in a week.
What is the actual constraint? If it is time-to-interactive on mid-range phones, that is a bundle and main-thread problem, and no framework choice fixes it by itself.
The number I'd bring to keep the discussion grounded: React leads every developer survey at around 45% of respondents, and appears on under 8% of actual websites, against roughly two thirds running jQuery. Those measure different populations — who answers surveys versus what is deployed — and confusing them is how teams end up believing the whole industry has already moved.
Where I would push back hardest is on rewriting a working application to change framework. That trades a known system for an unknown one and delivers no user-visible value. If the complaint is performance, measure first — the cheapest wins are almost always image priority hints, image dimensions and back/forward cache eligibility, and most of the web has not made them.
What is being tested
Whether you treat framework choice as a constrained decision rather than a preference, and whether you know the survey-versus-deployment gap.
Angular, Vue, Svelte and Solid all adopted signals. React shipped a compiler instead. What is the trade-off?
What you said
A strong answer
It is the cleanest architectural fork in modern frontend, and both sides are defensible.
Signals give you precision. A signal knows exactly which computations depend on it, so an update touches only what actually changed. No diffing, no reconciliation, no memoisation hints. It is predictable in the strong sense: you can reason about what will re-run.
The cost is that reactive ownership leaks into every API you touch. You now have to know whether a value is a signal, whether reading it inside this scope creates a subscription, whether passing it to a function preserves reactivity or flattens it. That is a real, permanent cognitive tax, and it is why signal frameworks have long documentation sections about when reactivity is lost.
React kept components as pure functions of props and state and moved the optimisation into the build. The compiler reached 1.0 in October 2025 and does automatic memoisation, so the manual hint APIs stop being something you hand-place. The mental model stays simple: given these inputs, render this output.
The cost there is an opaque build step you have to trust. When it does not memoise something you expected, you are debugging a compiler's decision rather than your own code. And its published gains are vendor-measured — around 12% on loads, more on specific interactions.
What I'd add if asked where this is going: the standards proposal for signals is still at stage 1, and the proposal's own authors estimate at minimum two to three years after standardisation before native browser availability. So signals are a framework feature for the rest of this decade, not a platform feature. Anyone architecting around imminent native signals is planning against a date that does not exist yet.
What is being tested
Whether you can hold two valid designs in mind and articulate the cost of each, rather than defending the one you use.
React Native, Flutter or native — how do you decide?
What you said
A strong answer
I'd decide on three things, and code sharing is not one of them.
First, what is the incident response time you need? This is the underrated one. If a production bug means submitting to review and waiting — realistically 24 to 72 hours for an update, up to a week for a first submission — then your mean time to recovery is set by a queue you do not control. React Native's real advantage is over-the-air JavaScript updates, which moves that back into your own pipeline. That is an architectural argument, not a convenience one, and it is a better reason to choose it than code sharing.
Second, does it need to feel native or look identical? Flutter draws every widget itself, which is exactly why it is pixel-consistent across six platforms and exactly why accessibility, text selection and platform idioms are approximations. Design-led product where brand consistency wins: Flutter. Product where users should not be able to tell it is cross-platform: native, or Kotlin Multiplatform.
Third, who is the team? A JavaScript team on Kotlin Multiplatform will suffer — the on-ramp is Gradle and native tooling. Two strong native teams tired of writing the same networking layer twice are exactly its audience, and its distinctive property is that it does not force a shared UI: share the logic, write SwiftUI and Compose natively.
Two things I'd correct if they came up. React Native's New Architecture is not optional or pending — the legacy components were removed in February 2026. And the widely quoted Flutter-versus-React-Native adoption split comes from fieldwork that is several years old and re-quoted annually. The honest picture is that Flutter leads on volume of new apps shipped and React Native leads on revenue-weighted and enterprise presence.
And the branch people skip: if the app is fundamentally a website with a launcher icon, build a website. That is right more often than anyone admits.
What is being tested
Whether you reason from incident response and team composition rather than from a code-sharing percentage, and whether your facts are current.
Our Lighthouse score is 95 but users say the site is slow. What is happening?
What you said
A strong answer
Almost certainly the gap between lab and field, and there is a specific pair of numbers that captures it.
Median total blocking time across the web is 92 milliseconds on desktop and 1,916 milliseconds on mobile. That is a twentyfold difference on the same pages. Your lab tool runs on your machine; your users are on a mid-range phone on a congested network. A high lab score and unhappy users is the expected result, not an anomaly.
And note which metric is being missed. Lab tools are good at the loading metric and poor at interaction latency, because interaction latency is a property of real user behaviour — it is the worst interaction across the whole visit, and a synthetic run does not interact with your page the way a person does.
So I'd get field data first — the interaction metric at the 75th percentile, segmented mobile and desktop — and attribute it with the Long Animation Frames API rather than guessing at which handler is slow.
There's also a structural blind spot worth naming: if this is a single-page application, field data currently measures one load at the hard navigation and nothing thereafter. Every route change your users make is invisible. The Soft Navigations API fixes this and is arriving unflagged in Chrome 151; until then, a single-page app's field numbers describe its first paint and nothing else.
Then I'd check the cheap things before touching architecture, because the data says most sites have not: about 83% of mobile pages do not set a priority hint on their largest image, 62% have images with no dimensions, 23% disable back/forward cache with a no-store header, and around 16% lazy-load the very element the loading metric measures. Every one of those is a one-line change.
What I would not do is propose server rendering as the fix. It helps the loading metric and does nothing for interaction latency — that is main-thread work, and a server-component-heavy app can ship more JavaScript than the single-page app it replaced.
What is being tested
Whether you distinguish lab from field, know which metric each is bad at, and reach for measurement before architecture.
What does exactly-once actually guarantee in a streaming pipeline?
What you said
A strong answer
Less than the phrase implies, and being precise about it is the whole answer.
Exactly-once is a property of a closed system. Inside a streaming platform's own boundary, you get it through four mechanisms that people constantly conflate: an idempotent producer deduplicating by producer, epoch and sequence per partition, per session; transactions making a multi-partition write atomic with the consumer offset commit; read-committed consumers reading only to the last stable offset; and fencing that bumps the epoch to stop a zombie instance.
What none of that gives you is exactly-once into a sink outside the system. A stream processor can give you exactly-once state via checkpointing; end-to-end delivery additionally requires a two-phase-commit sink — and that sink only reveals output at checkpoint boundaries, so you have bought correctness with latency. An arbitrary non-idempotent side effect, like charging a card or calling a third-party API, cannot be made exactly-once by any engine.
So the useful question is never ‘does it support exactly-once’. It is ‘exactly-once between which two points, and what does the sink do?’ In practice the answer is usually: at-least-once delivery plus an idempotent sink, which is a perfectly good design and should be described honestly rather than dressed up.
One operational detail worth adding, because it shows you have run this: a hanging transaction pins the last stable offset, which stalls every read-committed consumer and stops log compaction. Recent broker versions bump the producer epoch per transaction and enable server-side defences by default, which closes that hole — but if you are on older clients you can still get there, and the symptom is consumers that have stopped without any error.
What is being tested
Whether you can state the guarantee boundary precisely instead of repeating a marketing term.
We bought a data observability platform. Do we still need tests?
What you said
A strong answer
Yes, and the reason is structural rather than a matter of belt-and-braces.
Observability detects four things well: freshness, volume, schema change and distribution shift. Those are genuinely valuable and genuinely automatable — the pipeline that silently stopped, the upstream column that changed type, the day the null rate tripled.
There is an entire class it structurally cannot detect: semantically wrong data at stable volume. A join grain that quietly went from one-to-one to one-to-many, so revenue doubles. A new region writing local currency into a column everyone assumes is dollars. A filter dropped in a refactor, returning test accounts to a production aggregate. In all three cases the row count is normal, the schema is unchanged and the distribution barely moves. Nothing statistical fires. The number is simply wrong and somebody makes a decision on it.
The only defence against that class is a human assertion about meaning — a uniqueness test at the expected grain, a contract pinning the currency, a reconciliation to a total you can verify independently. So a team that bought the platform and let its test suite rot has moved its coverage backwards while feeling more covered.
I'd also raise the operational failure mode, because it is the one that actually bites: alert fatigue. Anomaly detection on every column of every table generates more alerts than anyone reads, and a channel nobody reads is worse than no channel — it converts a real incident into a scrolled-past notification. I'd instrument the tables that feed decisions, deliberately leave the rest unmonitored, and say so out loud so the gap is a stated choice rather than an oversight.
And on contracts, if they come up: most contract tooling does not enforce contracts. It authors them, validates the document and generates artefacts. The question that finds the truth is ‘what happens, mechanically, when a producer violates one?’ If the answer is ‘an alert fires’, that is monitoring with better documentation.
What is being tested
Whether you know the detection boundary of statistical monitoring and can name the class of failure it cannot reach.
How would you measure whether AI has made your engineering org faster?
What you said
A strong answer
I'd separate three things that get conflated, and I'd lead with the strongest available evidence rather than an opinion.
First, what people believe versus what is measured. The best-documented trial in this area put sixteen experienced developers on real issues in large repositories and found they were 19% slower with AI — while believing they had been sped up by 20%. That gap between self-report and measurement is the finding I'd carry into any productivity discussion. But I'd cite the update, not the headline: in early 2026 the same team said their follow-up data was unreliable, because 30–50% of participants were declining to submit tasks they did not want to do without AI. Their revised estimate is roughly an 18% speed-up with a confidence interval that crosses zero. Anyone still quoting the 19% figure is quoting something the authors have publicly qualified.
Second, individual speed versus system throughput. The large delivery-research programme found AI adoption significantly improves individual productivity, flow and satisfaction and negatively affects delivery throughput and stability. Those are not contradictory — removing the coding bottleneck saturates everything downstream of it: review, testing, release. Their framing is that AI is an amplifier of the existing system. So the question I'd actually ask the org is not 'should we adopt this' but 'what is our next bottleneck, and have we widened it first?'
Third, what I'd instrument. Not lines or acceptance rate — those are activity and Goodhart applies immediately. I'd watch change failure rate and time to restore alongside throughput, because the research predicts precisely that those diverge, and if they do I know where to invest. I'd also watch review latency and queue depth specifically, since that is the bottleneck the mechanism predicts.
And I'd plan for a dip. The same programme publishes on the productivity dip during rollout. Leaders who don't expect it read it as failure and reverse a decision too early.
What is being tested
Whether you know the study was revised, can separate individual speed from system throughput, and reason from the bottleneck mechanism rather than a productivity number.
How do you prioritise a roadmap?
What you said
A strong answer
I'd use a framework, and I'd be honest with the room about what it is doing.
What these frameworks actually are. RICE, ICE and WSJF are alignment artefacts, not decision procedures. Their value is forcing assumptions to be explicit, exposing where stakeholders disagree, and leaving an auditable record — none of which requires the arithmetic to be sound, which is fortunate. There is no empirical evidence that any of them improves outcomes. The peer-reviewed prioritisation literature doesn't even evaluate them; it evaluates AHP and cumulative voting.
The specific defects, if pressed. RICE multiplies an ordinal impact ladder as though it were a ratio scale and then divides, so the output rank ends up dominated by whichever input has the widest dynamic range — usually reach. And its confidence term double-counts uncertainty that is already in the reach and impact estimates, which means it discounts hardest exactly the high-variance ideas that produce outliers. WSJF is the best-grounded of them — it's the weighted-shortest-processing-time rule — and its grounding is the one most often invalid, because that rule assumes a single machine, no dependencies and known durations, and a product backlog violates all three.
The case that changed how I think about this. At a major search engine, an ad-title display change was stack-ranked low and sat in the backlog for six months, until an engineer built it in a few days on his own initiative. It raised revenue 12% — over $100M a year in one market, worth more than a hundred person-years of that org's normal output. No scoring system would have caught it, because an untested large effect looks exactly like low impact and low confidence from the outside.
So the lever isn't a better score. It's making the cost of trying a low-ranked idea small enough that low-ranked ideas get tried — which is an argument about cycle time and experimentation infrastructure, not prioritisation method. The return on reducing the cost of being wrong exceeds the return on being right more often.
What is being tested
Whether you can use a framework while knowing what it cannot do, and whether you understand that scoring systems structurally cannot find outliers.
What do you make of the Spotify model?
What you said
A strong answer
It was never fully implemented at Spotify, and the interesting part is which bit failed.
On the provenance. A product manager who joined in 2017 reported that the squad model was 'only ever aspirational and never fully implemented'. An agile coach who was there when the paper was written said: 'Even at the time we wrote it, we weren't doing it… People have really struggled to copy something that didn't really exist.' And a co-author said it worried him that people treated it as a copyable framework. The write-up was planned as a series — autonomy made the first cut and the parts on alignment and accountability were never written. The industry copied a third of a design.
What actually failed was not the cross-functional team. It was the matrix management of engineers. Chapter leads managed by specialism and had little responsibility beyond career development, so the product manager had no engineering peer — nobody accountable for delivery who could negotiate priorities at the same level. Concretely: a team with backend, web and mobile engineers had at least three engineering managers who might need to be involved in one disagreement, escalating to a director if they couldn't agree.
The line I'd take into any org-design conversation is theirs: 'Every responsibility a team cedes to increase its focus becomes a new cross-team dependency.' Autonomy isn't free — it converts internal coordination into external coordination. Spotify never defined a common process for cross-team collaboration, so every team having a unique way of working meant every collaboration needed a unique way of engaging, and overall productivity suffered.
Which generalises to the thing I'd actually say: coordination cost is conserved. You pay it inside a team through hierarchy or across teams through dependencies. So the question about any structure isn't 'is this agile' — it's 'where does this move the coordination cost, and who owns it once it moves?' The Spotify story is what happens when the answer to the second half is nobody.
What is being tested
Whether you know the model was aspirational, which component failed, and can generalise it into a usable principle rather than a debunking.
Are structured interviews really better, and by how much?
What you said
A strong answer
Yes, and the numbers everyone quotes are the wrong ones.
The 1998 meta-analysis is still on every hiring slide, and it was substantially corrected in 2022. The correction was about range restriction — the statistical adjustment for only observing job performance among people you actually hired — and the old assumptions produced significant over-correction. Most estimates dropped by ten to twenty points.
What moved: work samples fell from about .54 to .33, the largest drop, having been the top-ranked method for two decades. Cognitive ability fell from about .51 to .31 and was later revised lower still — the authors say it is no longer the stand-out predictor. Unstructured interviews fell to about .19, with a credibility interval whose lower bound is below zero. And years of experience fell to roughly .07, which is close to nothing — worth saying out loud, because 'five to seven years' is the most common filter in the market.
Structured interviews became the top-ranked method at about .42, having fallen the least.
But the part nobody quotes is the part that should change what you do. The authors state it explicitly: structured interviews are '.42, plus or minus .24' — an 80% credibility interval from .18 to .66 — and they note that conveys a different flavour than a single number. They even suggest comparing methods on the low end of the interval, where structured interviews lose to empirically-keyed biodata despite winning on the mean.
Which means the method is worth far less than the execution. A badly-run structured interview and a well-run one differ by more than the gap between structured interviews and most of the list. So I'd spend the effort on rubrics written before seeing candidates, independent scoring before the debrief, and interviewer calibration — not on choosing a method.
And I'd be honest that the correction is contested. A 2023 paper disputes the range-restriction argument underneath it, and roughly four-fifths of the input studies are the design in question. I'd quote the new numbers, give the interval, and say the dispute exists.
What is being tested
Whether you know the figures were revised, quote the interval rather than the point, and draw the conclusion that execution beats method selection.
How should we stop candidates cheating with AI?
What you said
A strong answer
I'd start by retiring two beliefs that the evidence does not support, because both are load-bearing in most people's plans.
First: 'add a twist to a known question' does not work. In a controlled experiment where candidates were instructed to cheat, verbatim public problems had a 73% pass rate and lightly modified versions 67% — statistically indistinguishable. Only genuinely custom questions, with novel inputs and outputs, moved the number, to 25% — which is below the honest control group at 53%. So a truly novel question is harder with a chatbot than a public one is honestly.
Second: 'an experienced interviewer can tell' is not supported. In that same experiment, not one of thirty-two interviewers raised a cheating concern, despite being prompted several times to flag problems — while rating themselves confident in about 72% of their decisions. Around 81% of the cheating candidates reported no worry about being caught. Detection is not a control; it is a feeling.
So what actually works is question construction, not policing. Genuinely custom problems. Longer, multi-part, more ambiguous questions that are problem-solving heavy rather than coding heavy. And — the clearest convergence across companies — comprehension over production: reading and debugging an existing multi-file codebase rather than writing a function from a blank editor. One employer's stated rationale is simply that engineering work involves far more code reading than code writing, which is true regardless of AI.
The other real option is to stop fighting it. One company now expects AI use and replaced its fundamentals screen with an AI-assisted competency scored on problem-solving, ownership of the code produced, and collaboration effectiveness. Their most interesting published finding: candidates with minimal AI experience often struggled — not on coding, but on the judgement to guide the model and notice a poor suggestion.
Whatever we choose, I'd state it per round and in advance, which is what the companies handling this best do. And I'd note that not one interviewer at a large tech company in the survey said their employer had dropped algorithmic questions — the algorithmic interview is not dead, it is just constructed differently.
What is being tested
Whether you know the detection evidence, avoid the twist fallacy, and reach for question construction rather than surveillance.
You are told a company's loop is five rounds. What do you actually need to find out?
What you said
A strong answer
The round count is the least informative thing about a loop. I'd want four other things, and most are askable.
1. What each round assesses, and who runs it. The published loops that exist differ enormously in composition — one company splits coding, algorithms, system programming and architecture into separate rounds; another runs three forty-five minute rounds total. Knowing 'five rounds' tells me nothing about which of those it is. And a values or cross-functional round run by someone from a different department is assessing something quite different from a hiring-manager round.
2. What the pre-loop gate is, because it signals the level I'm actually being considered at. Across the published loops there is a consistent pattern: mid-level is gated by an automated assessment, senior by a human senior leader, and manager adds a written artefact. If I'm being screened by a machine for a senior role, something is off.
3. The AI policy, per round, in writing. This genuinely differs three ways right now — forbidden, expected, and depends-by-round — and there is at least one live case where widely-syndicated press reporting says a company allows AI while its own candidate page says it does not. Both can be true if a pilot is narrow. Asking the recruiter is the only thing that binds, and no reasonable employer minds the question.
4. Whether any round is in person. The 'everyone is going back to the office' narrative is overstated — one large company says it is selectively reintroducing in-person final rounds after virtual screens; another states its process is entirely virtual. That is a logistics question with a real answer.
And one I'd ask for my own sake rather than for preparation: 'what would make you regret hiring someone into this role?' It is hard to answer from a script, and the hesitation is as informative as the answer.
What is being tested
Whether you know that composition and the gate matter more than the count, and whether you verify AI policy at source rather than from press coverage.
Design a URL shortener.
What you said
A strong answer
I'd start with numbers rather than boxes, because every decision after that is either justified by a number or it is a preference.
Five things I'd ask for: daily new links and peak concurrency, the read-to-write ratio, payload size, the latency target and at which percentile, and what is allowed to be stale. Two minutes, and they convert the rest of the hour into arithmetic.
Then the arithmetic, out loud. A hundred million new links a year at ~500 bytes of metadata is 50 GB a year. Ten years fits on one ordinary database instance with room to spare. So I'd say plainly: this fits on one machine for a decade and I am not going to shard it. Reaching for a sharding scheme here is the most common way a candidate signals junior.
The read-to-write ratio is enormous — perhaps 10,000:1 — so the entire read path is a cache lookup and the database is almost never touched. Everything else follows from that one number.
The only genuinely hard part is ID generation. A random 7-character key over 62 characters gives ~3.5 trillion possibilities, but random generation needs a uniqueness check, which is a read before every write. A counter encoded in base 62 removes the check and leaks your volume to anyone watching the keys increment. Both are defensible; I'd say which I picked and why.
And the part almost nobody raises unprompted: a URL shortener is an open redirect, so it will be used for phishing within a week of launch. Scanning, per-account rate limits and a takedown path aren't a nice-to-have — raising them is what moves you from 'built one' to 'operated one'.
What is being tested
Whether you establish numbers before drawing, choose the boring option deliberately, and raise abuse without being prompted.
A dependency slowed down and now every request is failing, including ones that never touch it. Why?
What you said
A strong answer
That is a resource-exhaustion cascade, and the mechanism is worth being able to state precisely because it predicts the fix.
Little's Law: concurrency equals arrival rate times latency. When the dependency's latency rises, your in-flight request count rises with it — same traffic, longer holds. Eventually you exhaust the connection pool or the thread pool, and at that point every request fails, including ones with no relationship to the slow dependency. The failure surfaces somewhere unrelated to the cause, which is why it is confusing at 3am.
Retries are the accelerant. Naive retry turns a degraded dependency into a denial of service against it, and it multiplies per layer — three layers retrying three times each is twenty-seven times the load at the bottom.
The controls are unglamorous and they work. A timeout shorter than your caller's timeout. Bounded concurrency per dependency, so one slow thing cannot consume the whole pool — this is the one that actually contains the blast radius. A circuit breaker so you stop calling something that is failing. Exponential backoff with jitter, because synchronised retries reconverge into a thundering herd. And load shedding, which is the deliberate choice to fail some requests fast rather than all of them slowly.
The question I'd ask on any code path in review: what is the timeout on this call, and what happens when it fires? A surprising number of production calls have no timeout at all, which means the answer is 'wait forever'.
And I'd note the related capacity point: queueing time scales with 1/(1−utilisation), so going from 90% to 95% utilisation doubles latency while looking like a five-point move on a dashboard.
What is being tested
Whether you can explain the cascade mechanism rather than just naming circuit breakers, and whether you know retries make it worse.
How do you decide what to block in a code review?
What you said
A strong answer
I separate blocking from optional explicitly, and I prefix the comments, because most review friction is authors guessing which comments are requirements.
I block on: correctness, security, data loss, an interface I cannot take back once it ships, and anything that will page someone at three in the morning.
I do not block on: naming I would have chosen differently, structure that is merely not my preference, or anything a formatter or linter could decide. If a machine can settle it, a human should not be spending review capacity on it.
Two comments catch most of the real bugs, and both are cheap enough to leave on a change I am not the ideal reviewer for. 'What happens the second time this runs?' — retries, redeliveries and double-clicks are everywhere, and non-idempotent handlers are the most common serious defect that passes review. And 'what happens if this is empty, null, or enormous?' — the three inputs nobody tests and the three that turn up in production within a month.
I ask rather than assert where I might be wrong. 'What happens if this is empty?' costs nothing when I'm wrong; 'this breaks on empty' costs credibility.
And a rule about the review culture rather than the review: if I find myself leaving the same comment repeatedly, it is not a review comment. It is a lint rule, a type, or documentation I have not written yet. Reviews that keep re-teaching the same thing are a signal that the fix belongs somewhere else.
What is being tested
Whether you reserve blocking for what is irreversible, and whether you notice that a repeated comment is a missing rule.
How would you know whether the feature you shipped worked?
What you said
A strong answer
I'd want three things, and the second is the one most people skip.
A primary metric tied to the user's problem, not to the feature. If we shipped a search improvement, the primary is successful sessions — a search followed by a click and no immediate re-search — not searches performed.
A guardrail that would catch the feature succeeding at something else's expense. For that search change, latency and zero-result rate: the easy way to improve apparent relevance is to return more things more slowly. Naming the guardrail unprompted is the single highest-value sentence available in this round.
And a window, stated in advance. Two weeks, because week one is novelty — and I'd be careful there, since the people who built the largest experimentation platforms report that novelty effects are actually uncommon, mostly showing up in recommender diversity and finite-resource treatments. Before I attribute a decay to novelty I'd check for a sample ratio mismatch and whether the analysis population changed mid-flight.
What I would not use is adoption on its own. Usage tells you people found it, not that it helped, and a feature can be heavily used and actively bad while the usage chart looks like success the entire time.
And I'd say what I'd do differently at each outcome before running anything. If there is no answer to 'what changes at 85% versus 95%', the number is decoration and I'd rather find that out before building the instrumentation than after.
What is being tested
Whether you pair a primary metric with a guardrail, state a window, and can say what decision the number would change.
You think the requirement is wrong. What do you do?
What you said
A strong answer
I raise it before building it rather than being right about it afterwards — but I'm careful about which half of that is my call.
First I check I actually understand the problem, because 'this requirement is wrong' is frequently 'I do not know why this was asked for'. I'd go to whoever wants it and ask what they are trying to achieve, not what they want built. Very often the answer reveals a constraint I could not see.
Then I make the trade-off visible rather than refusing. 'We can do that. It pushes the migration to Q3 — do you want that trade?' converts a refusal into a decision and hands it to the person who owns the priority, which is almost never me.
And I bring the smaller version. A no with an alternative is a negotiation; a bare no is a wall. 'This is six weeks, and here is a two-week version that covers the main case' is the most useful sentence I know in this situation.
Where I'd stop: if they hear the trade-off and still want it, I build it. Overreach is the failure mode nobody warns you about — an engineer who blocks a small requested change on a large hypothesis becomes expensive to work with, and the credibility you spend there is not available next time when you are right about something that matters.
The version of this I'd flag as genuinely non-negotiable is different: correctness, a legal or safety obligation, or something irreversible. That is not a preference disagreement and it should be escalated rather than traded.
And I'd write it down either way. Six months later somebody proposes the rejected option again, and the document is the only thing that stops the argument being re-run from scratch.
What is being tested
Whether you surface the trade-off rather than refusing, bring a smaller option, and know where your authority stops.
You've inherited a system nobody understands and something is broken. Where do you start?
What you said
A strong answer
With a search strategy rather than with the code, because I do not have the context that would make reading it efficient.
What changed. Systems that worked and then did not almost always had something change — a deploy, a config flip, a feature flag, a dependency upgrade, a certificate expiring, a data volume crossing a threshold, a traffic jump. A timestamp plus a deploy log resolves a large share of incidents before anyone opens an editor.
The blast radius, immediately. One user, one tenant, one region, one instance, or everyone? That single answer eliminates most hypotheses at once — a single-region failure is not a code bug, and a single-instance failure is not a deploy.
Then I bisect rather than stare. Client or server? Before or after the proxy? Read path or write path? Each halving is worth an hour of reading code, and I work the request path in order confirming each hop rather than assuming it — most wasted incident time is spent debugging a layer that was fine.
I write the timeline as I go, not for the postmortem but for me. Ten minutes in, nobody remembers what has already been ruled out, and the commonest waste is checking the same thing twice while the real cause sits unexamined.
One discipline I'd name explicitly: when the evidence contradicts my model of the system, the model is wrong. Time spent explaining away the evidence is time lost — and on a system I did not write, my model is a guess.
And I'd be careful about the first plausible correlation. A deploy five minutes before the errors is a correlation. I'd roll it back to restore service, and record that the diagnosis is unconfirmed — because a rollback that fixes the symptom is very often taken as proof and closes the investigation, and the signature of that mistake is the same incident recurring weeks later with a different deploy blamed.
What is being tested
Whether you have a halving strategy for a system you do not know, and whether you separate restoring service from diagnosing.
How do you find out what users actually need?
What you said
A strong answer
By asking about the past, never the future — that single rule does more work than any method choice.
'Would you use a feature that did X?' produces a polite yes with no predictive value. People are not lying; they are genuinely poor at forecasting their own behaviour, and they are being agreeable to someone who clearly wants a yes. 'Tell me about the last time you needed to do X — walk me through what you actually did' produces evidence, because it is a memory rather than a prediction. The workarounds people describe are usually the most valuable thing in the whole study.
Then I'd match the method to the question. Usability testing with five to eight people finds most usability problems and is the cheapest high-value thing available — but it says nothing about prevalence, which is what analytics is for. Analytics tells you where people drop off and never why. Interviews tell you what people currently do and what is hard about it. Surveys only measure the answers you already thought to offer, so they are for sizing something you have already characterised, not for discovering it.
The pairing that resolves most arguments: analytics for where, qualitative for why, and neither substitutes for the other. A team with only analytics knows exactly which step loses people and has no idea what to change.
And on feature requests specifically: a user asking for a feature is reporting a solution they designed. I'd ask what they were doing when they wanted it. The problem underneath usually admits a better answer, and occasionally reveals the feature would not have helped at all.
What is being tested
Whether you ask about past behaviour rather than stated intent, and whether you know what each method structurally cannot tell you.
Walk me through a project in your portfolio.
What you said
A strong answer
I'd structure it around decisions rather than outcomes, because a finished screen tells you someone can produce a finished screen and says nothing about whether I can be trusted with an ambiguous problem.
What the problem was and how I knew — the evidence, not the brief. What constrained me — technical, legal, timeline, an existing pattern I could not break. What I tried that did not work. What shipped. And what I learned afterwards that would change it.
The third item is the one that separates candidates and almost nobody includes it. A portfolio in which every decision was correct first time reads as a portfolio that has been tidied, and interviewers discount the whole thing accordingly. Showing the version I rejected, and why, is worth more than showing a better final frame.
I'd be precise about my own contribution. Team projects are expected; ambiguity about who did what is the single most common reason a portfolio falls apart at reference stage.
And I'd bring something that shipped and was measured if I have it. 'This reduced support tickets about X by a third' does more for me than any amount of visual craft, because it is evidence the design worked rather than that it looked right.
The thing I'd avoid is narrating the process as though it were linear. Real projects have a point where the original framing turned out to be wrong, and describing that moment honestly is more persuasive than a clean arc.
What is being tested
Whether you present decisions and rejected alternatives rather than polished outcomes, and whether you are precise about your own role.
How would you design a feature whose core is a model that is sometimes confidently wrong?
What you said
A strong answer
The design problem is that the output is plausible whether or not it is correct, so the interface has to carry the uncertainty the words do not.
The pattern that consistently works is: the model drafts, the human directs. Put the output in an editable surface as a starting point, so the user's effort goes into shaping rather than verifying. Almost every durable AI feature has this shape.
The second that works: the model narrows, the human chooses. Several options with what distinguishes them, rather than one answer. That is honest about uncertainty by construction and it is far easier to evaluate than a single confident claim.
The one that fails is the one most products reach for — the model decides and the human approves. A reviewer facing a queue of confident, plausible suggestions under time pressure approves nearly all of them, and the resulting near-zero override rate looks like model quality when it is a control that has stopped operating.
So my design test is: what would make a user disagree, and is disagreeing as easy as agreeing? If approving is one tap and disagreeing means leaving the flow or writing a justification, I have built rubber-stamping into the interface and no amount of training will fix it.
Other things I'd hold to: show the source when there is one, because a user who can check in one click will. Make the cost of a wrong answer visible — a dismissable suggestion needs no ceremony, one that sends an email needs a confirmation naming what will happen. Don't fake confidence or humility, since neither reflects a calibrated internal state; surface the genuine signals you have, like retrieval finding nothing.
And design the failure state first. What the user sees when the model returns nothing useful, or is slow, or is down — that is a large share of real sessions and it is almost always designed last, if at all.
What is being tested
Whether you design for uncertainty structurally, and whether you know why 'human approves' fails as an oversight pattern.
Nobody is using the design system. What went wrong?
What you said
A strong answer
Almost certainly that using it is slower than not using it, and that is a design problem with the system rather than a discipline problem with the teams.
The failure mode is a component library that is harder to find, learn and bend than writing a one-off. If contributing a variant takes three weeks of review, people fork — and every fork is a decision re-made badly, which is exactly what the system existed to prevent.
So the metric is adoption, not component count. A system with 200 components and 30% adoption is worse than one with 40 components everyone uses, and counting components rewards the wrong behaviour.
What I'd actually check: how long it takes to find the right component, whether the API bends to real cases or forces a fork, how long a contribution takes end to end, and whether the system's own examples match the current version. Stale examples are a quiet killer — people copy them and then diverge.
The strongest argument for the system is usually accessibility, and it is worth leading with. Getting focus management, keyboard interaction and screen-reader announcement right in a date picker is genuinely hard; getting it right once is achievable, and getting it right in every team's own date picker is not. That is the clearest case of 'decide once' paying for itself.
And the risk nobody names when adopting one: a system with inaccessible components actively scales the problem. It is the same leverage pointed the wrong way.
What is being tested
Whether you treat low adoption as a system defect rather than a compliance problem, and whether you know accessibility is the strongest case for it.
A prospect asks for a demo on the first call. What do you do?
What you said
A strong answer
I'd do a short one and buy the discovery first, because demonstrating before understanding produces a tour of features rather than answers, after which the customer politely disengages.
What I'd ask, in about ten minutes: what they are doing today — the actual workflow, who touches it, how long it takes, not their strategy. What made them start looking now, because there is always a trigger: an incident, an audit finding, a renewal, a regulation, a person who left. No trigger usually means no deal, however interested they sound, and finding that out on call one is a gift rather than a loss. Who else has to agree and what they care about. What they have already tried — this is the question that prevents the worst outcome, which is demonstrating an approach they rejected eighteen months ago for reasons I never asked about.
And the best question in the set: what would make this a failure in six months? It surfaces the real constraint and almost nobody asks it.
Then the demo is not a tour. Three things that solve the problem they just described, in their vocabulary, with their data shapes. Every screen that is not about their problem costs attention I cannot get back, and finishing early is a strength — it signals I knew what mattered.
If they insist on the full tour, I'd give it and treat it as a qualification signal: a buyer who will not spend ten minutes on their own problem is often not close to a decision, and I'd want to know that before investing a cycle.
What is being tested
Whether you protect discovery under pressure, and whether you know that no trigger event usually means no deal.
The product genuinely cannot do what the customer needs. What do you say?
What you said
A strong answer
I say so, plainly, early — and that is not martyrdom, it is the entire asset of the role.
A technical person who confirms everything is indistinguishable from the salesperson, and the customer works that out within two meetings. From then on I'm treated as part of the pitch and my answers stop carrying weight, including the true ones. That is a much more expensive loss than one deal.
The commercial version is never a flat refusal. It is: here is the constraint, here is what a workaround would cost, and here is the nearest thing that does work. That keeps it a solution conversation rather than a rejection, and it keeps the deal alive on honest terms.
If the honest answer is that they should buy something else, I say that too. Customers move companies, they talk to each other, and they remember who told them the truth when it cost something.
What I will not do is commit to a roadmap item I do not control. 'That's on the roadmap' from a technical person is heard as a promise, and when it slips the trust lost is mine, not the product manager's.
And I'd surface it before the security review rather than during it — a gap I name is a constraint we are working around together; a gap their reviewer finds in week ten is a credibility event that usually ends the cycle.
What is being tested
Whether you understand that credibility is the asset being protected, and whether you can decline commercially rather than bluntly.
A customer has escalated and is furious about an issue you think is minor. What is going on?
What you said
A strong answer
When the severity does not match the reaction, the escalation is usually about something other than the incident, and treating it as a technical problem is why technically-perfect handling sometimes fails to calm one.
The usual causes: the champion staked their reputation on choosing us and is now exposed. A renewal is close and this is leverage. A previous issue was handled badly and this one is carrying the accumulated cost. Or someone senior asked a question nobody could answer.
None of those are fixed by the fix. So I'd ask the champion privately what is really going on, and help them with their problem — which is usually needing to show their management that the situation is controlled. A written summary they can forward, with what happened, what is being done and by when, is often worth more than the patch.
On the mechanics: acknowledge the impact before explaining the cause, because explaining first reads as defending however accurate it is. Give a next update time and hit it even with nothing to report — 'no progress, next update in two hours' sustains trust far better than silence followed by good news, since the silence is what convinced them nobody cared. Separate the workaround from the root cause explicitly and say which is which. And do not diagnose in front of them; thinking aloud sounds like uncertainty at the moment they need to see competence.
And never blame another team in front of the customer. They bought one company. Internal attribution is my problem, and airing it tells them the organisation cannot coordinate.
What is being tested
Whether you diagnose the escalation rather than the incident, and whether you know silence is what created it.
How do you run a proof of concept that actually concludes?
What you said
A strong answer
Entirely in the setup, because the failure mode is not failing — it is drifting. The champion moves on, and eight weeks of engineering time produce no decision.
Success criteria written down and agreed before starting. Specific, measurable, and few — three is plenty. If I cannot get agreement on what success looks like, I have learned something more valuable than the proof of concept would have told me.
An end date, treated as real. An open-ended trial becomes free consulting, and free consulting has no natural end.
What happens on success, agreed up front. 'Then we'll discuss next steps' is not an outcome. 'Then we move to a paid pilot with these two teams' is.
Their data, their environment. A proof of concept on sample data in my environment proves nothing anyone doubted — the real objections only appear when real data does.
A named owner on their side, or it is a project I am running alone and will lose quietly.
And the hardest discipline: be willing to conclude that it failed. A proof of concept that cannot fail is a demonstration with extra steps, and sophisticated buyers recognise it immediately. Concluding honestly that the fit is poor buys credibility that outlasts the deal — and the commercial version is 'this is not a fit for the use case we tested, and here is the one where it would be', which often keeps a smaller deal.
Watch for the drift signal: success criteria that quietly expand mid-trial are the customer telling you it is not working. That is a prompt for the honest conversation, not a reason to extend.
What is being tested
Whether you set an end date and an agreed next step before starting, and whether you can conclude a failure honestly.
Walk me through your pipeline.
What you said
A strong answer
I'd walk it stage by stage and I'd include the parts that do not flatter me, because a pipeline in which everything is progressing is not credible to anyone experienced.
My stages are defined by something the customer did, not by how I feel about it. 'They're interested' is not a stage. 'We've met the economic buyer and confirmed budget exists' is, because someone else can verify it.
So I'd cover: what is committed and why I am confident — meaning I will personally be wrong if it slips, not that I think it is 80% likely. What is best case and what specifically is missing, which is usually that I have not reached the person who signs. A deal I expect to lose and why. And anything that has not moved stage in two cycles, which is usually dead and inflating my coverage — I'd be taking it out rather than carrying it.
On coverage: the three-to-four-times rule of thumb is meaningless without my own historical win rate. Coverage is quota divided by win rate — if I close one in five, I need five times, and a borrowed multiple is somebody else's number.
The question I'd expect back is what made me move a deal backward, and I'd have a specific answer. Moving a deal forward is easy to describe; moving one back costs something, and a seller who has never done it is either extraordinarily lucky or is not applying the exit criteria.
Because the thing being assessed here is forecast discipline, not attainment. Attainment is partly territory. Forecasting is entirely me, and the whole company plans on that number.
What is being tested
Whether your stages are defined by customer actions, and whether you volunteer a loss and a stage regression.
What is the most common reason you lose deals?
What you said
A strong answer
Honestly: to 'do nothing'. Most enterprise deals are lost to inertia rather than to a competitor, and treating a rival as the opponent is how you lose to the status quo while winning the feature comparison.
Which changes the question I ask. Not 'why are we better than the alternative' but 'what happens to you if nothing changes?' If there is no cost to waiting, there is no deal this quarter however positive the meetings are.
The second most common is a stakeholder I never met. Technical enthusiasm does not survive a security review, a procurement process or a legal term nobody prepared for. So I'd want to know early who reviews security, what that process involves, how long it usually takes, and whether anything has failed it recently. Champions answer this readily and are often relieved to be asked.
The third is a champion with no political capital — genuinely enthusiastic, genuinely unable to get it funded. That is the hardest one to detect because the meetings go well.
What I'd do differently as a result: qualify the trigger event on the first call, map the approval path before the demo rather than after, and treat the security review as a sales stage with its own timeline rather than an administrative step at the end.
And I'd rather disqualify early than carry a deal. A pipeline full of deals that will never close is worse than a smaller honest one, because it makes the forecast fiction and it consumes the cycles that would have found a real one.
What is being tested
Whether you name inertia rather than a competitor, and whether you treat the approval path as discovery rather than paperwork.
An engineering team wants budget for a migration. How do you assess it?
What you said
A strong answer
I'd look for four things, and their absence is usually why a proposal comes back rather than the number being too big.
A named baseline. 'This will save money' is unfalsifiable. 'We currently spend X on this and this reduces it to Y' can be checked, and being checkable is what makes it credible.
Cost avoidance separated from cost reduction, and labelled. Reduction means a line goes down and I can see it in a budget. Avoidance means a line would have gone up. Both are real, only the first shows up in my numbers, and presenting avoidance as reduction is the fastest way to lose me permanently.
The cost of the change itself — engineering time at a loaded rate, dual running, migration risk. A business case with no implementation cost reads as naive, and it is the first thing I look for.
What happens if we do nothing. Every proposal competes against 'not yet', and only the do-nothing cost argues against that. If support ends on a date, say the date and say what we carry afterwards.
And I'd rather have a range with a stated downside than a point estimate, because I already know the point estimate is wrong and I am trying to work out by how much. A range answers the question I was about to ask.
One thing I'd raise from my side: whether this shifts capital spend into operating spend. Buying servers was capital — a large number once, spread over years. Renting them is operating, hitting profit immediately. The total can be similar and the profit line looks completely different, which is why finance sometimes resists a migration that is obviously right technically.
What is being tested
Whether you know what makes a business case credible to finance, and whether you can name the capital-to-operating shift.
Which contract terms should an engineering team actually care about?
What you said
A strong answer
Five, and the one they least expect is the one I'd push hardest on.
The breach notification window, as an engineering requirement. If we commit to seventy-two hours from awareness, someone has to detect it, someone has to decide it qualifies, and someone has to notify — and the on-call runbook should name who. A commitment nobody in the response path knows about is not a control, it is exposure with a signature on it.
Data residency, before a region is chosen. Retrofitting residency is a migration, and it is always discovered at the worst moment — usually during a customer's security review in week ten of a deal.
Sub-processor obligations, before adding a vendor. Every tool in the pipeline becomes one, and adding one can trigger a notice period to customers. Teams add vendors in an afternoon and the obligation surfaces months later.
What the service level actually measures. A monthly average hides a bad week, and 'excluding scheduled maintenance' can make a number meaningless. Engineers can usually tell within minutes whether a proposed measurement is gameable or unachievable, and that is far cheaper before signing than after.
And exit — which is almost never negotiated and is the thing you will most want. Export format, timeframe, and whether extraction costs anything, in writing. It is cheap to get at signing and impossible afterwards, and a vendor unwilling to commit is telling you what switching cost they are relying on.
The general point I'd make to engineering: legal is not an obstacle, it is under-informed. Most bad terms exist because nobody technical read the clause until it was a commitment, and reading it takes twenty minutes.
What is being tested
Whether you can translate contract language into on-call and architecture requirements, and whether you push on exit.
An engineer says they were passed over for promotion unfairly. How do you handle it?
What you said
A strong answer
I'd separate three possibilities before responding, because they look identical from the outside and need opposite answers.
One: they are not yet operating at the next level and nobody told them clearly. That is the most common, and it is a management failure rather than theirs. The test I'd apply is simple — if I asked their manager to name two specific moments this cycle where next-level behaviour was available and did not happen, could they? If not, the case was argued badly and the engineer is right to be aggrieved even if the outcome was correct.
Two: they are operating at that level and the evidence never reached the room. Calibration is a group of managers comparing people they mostly have not worked with, from written summaries. The quality of that summary is a large part of the outcome. Second-order work — whose design changed after a review, what the team stopped doing, what did not break — leaves no artefact of its own, so it vanishes unless someone deliberately wrote it down. That is a fixable process problem and I'd fix it rather than defend the result.
Three: budget or band position. Sometimes the answer is genuinely 'not this cycle' for reasons unrelated to performance, and the honest thing is to say so rather than manufacture a development gap to justify it. People detect that immediately and it is corrosive.
What I'd give them either way is the gap stated in the ladder's own terms — 'what would I need to demonstrate to be operating at the next level' is a fair question with a real answer — and a date to revisit.
And I'd note the trap they may be in: being so valuable in their current scope that moving them is expensive. That is a good problem and it does not resolve itself; someone has to decide to pay the cost.
What is being tested
Whether you separate a real gap from a documentation failure from a budget constraint, and whether you will say the uncomfortable third one.
How do you forecast, and when do you move a deal to commit?
What you said
A strong answer
Commit means I will personally be wrong if this slips, not that I think it is likely. That distinction is the whole of forecast discipline, and where commit is treated as 'probably', the forecast is noise the company still plans on.
What has to be true before I commit a deal: I have met the person who signs, not just the champion. Budget is confirmed to exist, not assumed. The approval path is mapped — security, procurement, legal — with a rough timeline for each. And there is a compelling event with a date attached, because a deal with no deadline slips indefinitely without anyone deciding to slip it.
Stage-weighting is better than a gut number and it is still a model. Multiplying each deal by its stage's historical close rate degrades badly when sellers park deals in a favourable stage, so it is only as honest as the stage definitions.
What I'd watch for in my own pipeline: aging by stage. A deal that has not moved in two cycles is usually dead and is inflating my coverage. Almost nobody runs that report, and it is the one that keeps the number real.
And the uncomfortable habit that makes the rest work: moving deals backward. It costs something to say in front of a manager, which is exactly why a seller who has never done it is not applying their own exit criteria.
What is being tested
Whether commit means a personal commitment rather than a probability, and whether you run aging by stage.
You have thirty seconds on a cold call. What do you say?
What you said
A strong answer
Not a pitch — a reason for them to spend the next two minutes, which is a different thing.
The shape that works is: why you, why now, and a question. Something specific that made me call this person rather than a list — a role change, a public commitment, a piece of hiring, a regulation that touches their sector. Then one sentence on the problem I usually find in that situation. Then a genuine question and silence.
What fails is opening with the product, because they have no reason to care yet, and opening with 'how are you today' because they know what it is.
I'd be honest that most calls end here and that is fine. The job is qualifying out fast as much as qualifying in — a cheerful 'not for us' in twenty seconds is a better outcome than a polite maybe that consumes three follow-ups.
And what I'm actually measured on is meetings that hold, not meetings booked. Those are different numbers and only one of them is worth anything. Volume targets on bookings reward spray, and the meetings that result get cancelled or wasted — so I'd rather make fewer calls with a real reason behind each than hit an activity number that produces nothing downstream.
What is being tested
Whether you lead with a reason specific to that person, and whether you know the metric is meetings that hold.
Procurement is demanding a discount you think is unreasonable. What now?
What you said
A strong answer
I'd separate what they are asking for from what they need, because a discount demand late in a cycle is often about process rather than price.
Procurement is frequently measured on savings achieved, so their job is to extract a concession — any concession. That means the question is not 'can we go lower' but 'what would let them record a win?' Payment terms, a longer commitment at the same rate, a bundled service, a ramp — several of those cost us far less than a straight discount and satisfy the same target.
What I would not do is discount without asking for something. A concession given freely resets the reference price for every renewal, and it teaches them that waiting produces a better number. If price moves, term or scope or timing moves with it.
And I'd check the discount is actually the blocker. Sometimes it is cover for something else — an unresolved security finding, a stakeholder who was never sold, an internal budget that has not been approved. Going straight to price on that is answering a question nobody asked and giving away margin for nothing.
If it is genuinely unreasonable, I'd be willing to lose it, and say so calmly with the reason. A deal signed on terms we cannot sustain becomes a bad renewal and a reference customer who feels overcharged.
What is being tested
Whether you trade rather than concede, and whether you check the discount is the real blocker.
How do you know a design worked?
What you said
A strong answer
By deciding what would count before it ships, because after launch every outcome finds an explanation.
A behaviour, not a perception. Satisfaction scores move slowly and for unrelated reasons; task completion, time to first success, error rate and repeat use move for reasons I can act on.
Tied to the user's problem rather than the artefact. If the problem was that people could not find their export, the measure is people finding it, not clicks on the new button. The button could be heavily used and the problem unsolved.
With a guardrail. Almost every design improvement can be gamed by making something else worse — a simpler flow that hides an option people need, a faster path that raises the error rate. Naming what I would not accept as a side effect is the part that separates a measurement plan from a hope.
And a segment. An aggregate hides the group the change hurt. A redesign can improve the median and be much worse for people using a screen reader or a small screen, and the top-line number will look like success the entire time.
The framework I'd reach for is goals-signals-metrics rather than a checklist of categories. The well-known five-category prompt is a way of articulating goals, and its own authors never said to fill in all five rows — teams that dutifully populate every category produce a dashboard rather than a decision.
What is being tested
Whether you pre-commit to a behavioural measure with a guardrail and a segment, rather than reporting satisfaction after the fact.
Engineering says your design is not feasible in the time available. What do you do?
What you said
A strong answer
Treat it as information rather than an obstacle, because it usually is — and my response depends entirely on which kind of infeasible it is.
First I'd find out what specifically is expensive. 'Not feasible' is rarely the whole design; it is normally one interaction, one state, or one piece of data that is not available where the screen needs it. That is a much smaller conversation and it often has a design answer rather than an engineering one.
Then I'd ask what the cheap version costs. If the expensive part is an inline live preview, is a confirmation step acceptable for now? I'd rather ship the 80% that solves the problem than hold the whole thing for a detail — and I'd want to know which 20% I am deferring, so it is a decision rather than a silent loss.
What I would not do is treat feasibility as someone else's problem. It is one of the four risks a product team owns jointly, and a designer who hands over a spec and waits is failing the same way an engineer who builds the spec without questioning it is.
Where I'd push back is on quality rather than scope. Cutting a state — the empty state, the error state, the loading state — is not scope reduction, it is shipping something unfinished, and those states are a large share of real sessions. I'd cut a feature before I cut a state.
And I'd write down what was deferred and why, because otherwise it is re-litigated in three months by someone who was not in the room.
What is being tested
Whether you find the specific expensive part rather than accepting the whole verdict, and whether you cut features rather than states.
How should a manager decide someone is not going to work out?
What you said
A strong answer
Long before the formal process, and the test is uncomfortable: if I asked the person today whether their job was at risk, would they say yes? If the answer is no and a formal process is three weeks away, the failure is the manager's, not theirs — and I'd say that to the manager before anything else happens.
What a fair process actually requires: the gap stated in specific observable terms, what 'fixed' looks like concretely, a genuine timeframe, real support rather than a paper plan, and no surprises at any point.
The most common mistake is a plan designed to document rather than to succeed. People recognise it immediately, it destroys trust with everyone watching, and the team always knows. If the decision has genuinely been made, the honest route is a conversation about an exit, not a performance plan run as theatre.
Before concluding it is the person, I'd rule out three things that look identical from outside and need opposite responses: a mis-hire, a mis-fit — right person, wrong role — and a management failure, meaning unclear expectations, an impossible remit, or no support. The diagnosis is the manager's job, and skipping it is how a company loses someone who would have been strong on a different team.
And the kindest version is also the fastest. Long ambiguous periods are worse for the person than a clear conversation, because they cannot make decisions about their own life while the outcome is undefined.
What is being tested
Whether you apply the no-surprises test, and whether you rule out mis-fit and management failure before concluding it is the person.
You do not know the answer. What do you actually do?
What you said
A strong answer
Say so in one sentence, then show the reasoning you would use to find out — what you would measure, what you would read, who you would ask. Interviewers are calibrating how you behave at the edge of your knowledge, because that is where you will spend most of the job. What loses the round is a confident wrong answer, because it tells them you will do the same thing in a design review.
What is being tested
Whether you can be uncertain out loud. Bluffing is detected far more often than candidates think, and it costs more than the missing knowledge.
The interviewer gives you a vague system design prompt. First move?
What you said
A strong answer
Establish scale, constraints and what “done” means before drawing anything. How many users, what read/write ratio, what latency budget, what consistency requirement, what is explicitly out of scope. A design without stated constraints cannot be evaluated, so an interviewer who watches you start drawing boxes has already learnt what they needed.
What is being tested
Whether you design under constraints or from a template. The template answer is obvious from the first minute.
“Tell me about your biggest failure.” What makes an answer good?
What you said
A strong answer
A real failure with a real cost, what you actually did about it, and the specific thing you changed afterwards — ideally something that later prevented a recurrence you can point to. The common failures are picking something trivially small, picking something that was somebody else's fault, or ending at the lesson without the change. The change is the whole answer.
What is being tested
Whether you learn from evidence or narrate around it.
What questions should you ask them, and which ones actually work?
What you said
A strong answer
Ask things whose answer would change whether you accept: what happens when a release breaks, who decides priority when two directors disagree, what the on-call load actually is in hours, what the last person in this role went on to do. Avoid questions the careers page answers. The strongest single question in most disciplines is a metric question — what do you measure and what is the current number — because most organisations cannot answer it.
What is being tested
Whether you are evaluating them. Candidates who ask nothing read as either uninterested or resigned.
You realise mid-answer that you have said something wrong. What now?
What you said
A strong answer
Correct it immediately and briefly, then continue. “Actually that is wrong — the reason is X” costs you a few seconds and gains you credibility. Interviewers who catch an error and watch you defend it downgrade you far more than for the error itself, because it is a preview of a code review with you.
What is being tested
Whether you update on evidence in front of someone. It is the single most-tested behavioural trait and almost nobody prepares for it.
How do you handle a take-home that would take two full days?
What you said
A strong answer
Ask what the time budget is meant to be, in writing. If they say four hours, spend four and write down what you would do with more — a README naming the shortcuts you took and why is often scored higher than the code. Silently spending twenty hours teaches them nothing about you at four, and it is the most common way strong candidates lose to weaker ones who scoped.
What is being tested
Whether you work to a budget and communicate scope. It is the same skill the job needs.
The interviewer is clearly wrong about a technical fact. Do you correct them?
What you said
A strong answer
Yes, but as a question rather than a contradiction — “I had understood it the other way, is that changed recently?” That gives them a route to update without losing face, and it is genuinely possible you are the one who is out of date. Letting a wrong premise stand means the rest of the discussion is built on it, and you will be marked down for the conclusions.
What is being tested
Whether you can disagree with someone senior without making it a contest.
A prospect says “send me pricing.” What do you do?
What you said
A strong answer
Find out what they are trying to do and what happens if they do nothing, before a number exists. Pricing sent into an undefined problem gets compared on price alone, which is the one axis where you have no advantage. If they will not engage on the problem, that is itself information about whether this is a real deal or a budget-benchmarking exercise for the incumbent.
What is being tested
Whether you qualify or react. Sending the deck is the reflex and it is usually the losing move.
Your pipeline is 4x quota. Is that healthy?
What you said
A strong answer
Not on its own. 4x coverage on a 25% historical win rate is break-even with no margin for slip, and coverage built from stage-one deals that nobody has qualified is not coverage. The honest questions are: what is the win rate on deals that reach stage three, how old is the average open deal, and how much of the number sits in one or two deals. Concentration is the risk that coverage ratios hide.
What is being tested
Whether you read pipeline as a forecast or as a comfort number.
You lose a deal you forecast as commit. What is the post-mortem?
What you said
A strong answer
Work backwards to the point where the deal was actually lost, which is almost never the last week. Usually it is an unvalidated champion, a decision process you never mapped, or a competitor who set the evaluation criteria before you arrived. The forecast error is the second problem: what did you believe, on what evidence, and what would have falsified it earlier.
What is being tested
Whether you can separate the deal loss from the forecasting failure. They have different fixes.
Your champion leaves the company mid-cycle. What now?
What you said
A strong answer
Treat the deal as re-opened rather than delayed. A single-threaded deal was always this fragile and the departure just revealed it. Re-qualify with whoever inherits the problem, and be honest with your own forecast immediately — the expensive mistake is carrying it at the same probability for another quarter because moving it looks bad.
What is being tested
Whether you single-thread, and whether you tell the truth in the forecast when it costs you.
Product cannot do what the customer needs. What do you say?
What you said
A strong answer
Say it, in the room, with what the workaround costs and when the real capability might land — and never invent a roadmap date you do not control. The deal you win by implying a capability becomes an escalation, a churn risk and a reference you cannot use. In technical sales your only durable asset is that the buyer believes you when you say yes, and that is created by the times you said no.
What is being tested
Whether you protect credibility over the current quarter.
How do you run a proof of concept that actually closes?
What you said
A strong answer
Define success criteria in writing before it starts, with the customer, and get agreement on who signs off that they were met. A PoC without written criteria never ends — it becomes free consulting that expands until someone loses interest. Scope it to the smallest thing that proves the contested claim, not the fullest demonstration of the product.
What is being tested
Whether you can say no to scope during a PoC. Most SEs cannot, and it is the main reason PoCs stall.
A customer is escalating loudly and is technically wrong. How do you handle it?
What you said
A strong answer
Separate the two threads. Acknowledge the impact and own the communication cadence first, because the escalation is usually about feeling unheard rather than about the technical point. Then correct the technical picture with evidence, privately if possible. Winning the argument in the escalation call and losing the account is a real outcome and it happens often.
What is being tested
Whether you can be right without making it the point.
What does a good health score actually measure?
What you said
A strong answer
Something that predicts renewal and that you can act on before it is too late — typically depth of use in the workflow that justified the purchase, not login counts. Most health scores measure activity, which correlates with usage rather than with value, so they go green right up until a champion leaves. The test of a health score is whether it ever caught a churn you would otherwise have missed.
What is being tested
Whether you can distinguish a leading indicator from a lagging one that feels reassuring.
The customer's architecture is bad. Do you tell them?
What you said
A strong answer
Yes, once you understand why it is that way — there is usually a constraint you cannot see, and leading with a redesign tells them you were not listening. Name the specific risk and what it will cost them when it fires, offer the smallest change that moves it, and let them choose. Architects who deliver verdicts get consulted less over time, which is the opposite of the job.
What is being tested
Whether you can be useful about a design you did not choose.
A customer's bill tripled and they are threatening to leave. First move?
What you said
A strong answer
Get the actual usage breakdown before any conversation about credits. Bills triple for three different reasons — genuine growth, a pricing or tier change, or a misuse pattern like a retry storm or an unbatched job — and they need three different responses. Offering a discount before you know which one it is fixes nothing and sets the price expectation permanently lower.
What is being tested
Whether you diagnose before you concede.
How do you decide what to escalate internally?
What you said
A strong answer
Escalate on what it would cost the business if you are wrong, not on how loudly the customer is asking. That means naming the revenue at risk, the timeline and the specific decision you need — an escalation without an ask is a complaint. Escalating everything spends the credibility you need for the one that matters this year.
What is being tested
Whether your escalations carry a decision. Most do not, which is why they get ignored.
Engineering wants to ship a feature that processes personal data of EU users. What do you ask?
What you said
A strong answer
What data, for what purpose, on what lawful basis, held where and for how long, and who else sees it. Then whether a data protection impact assessment is needed and whether any of it is a new processor relationship. The failure mode is being consulted at launch rather than at design, so the more useful answer includes how you get into the room earlier.
What is being tested
Whether you can turn a legal requirement into engineering questions rather than a blocking memo.
A vendor contract has an unlimited liability clause. The business wants to sign today.
What you said
A strong answer
Quantify the exposure rather than refusing on principle: what is the realistic worst case, is it insured, is it capped anywhere else in the agreement, and is this vendor genuinely irreplaceable this quarter. Then give the business a decision with a price on it. Counsel who only ever say no get routed around, which is worse for risk than counsel who say yes with a number.
What is being tested
Whether you convert legal risk into a business decision.
Engineering says cloud spend is “growth.” How do you test that?
What you said
A strong answer
Divide it by the thing that is supposed to be growing — cost per active user, per transaction, per unit of revenue. Genuine growth keeps the unit number flat or falling; waste shows up as a unit number climbing while the absolute number is explained away. The follow-up question is which of the top five line items has no owner, because unowned spend never falls.
What is being tested
Whether you reason in unit economics rather than totals.
What is the difference between opex and capex here, and why does anyone care?
What you said
A strong answer
Operating expenditure hits this year's profit; capital expenditure is spread across the useful life, so it lands lighter on the current year and sits on the balance sheet. That is why finance sometimes prefers a purchase to a subscription, and why capitalising engineering time on new capability is scrutinised — the rules are narrow and the temptation is not. It changes the shape of the P&L, not the cash.
What is being tested
Whether you understand why the same spend is argued about differently depending on its label.
A high performer is making the team worse. What do you do?
What you said
A strong answer
Name the behaviour and its effect with specifics, early, and treat it as a performance issue rather than a personality one — because it is. The cost of not acting is not the one person, it is the people who leave and the standard everyone else infers from what is tolerated. Managers usually know this and delay anyway because the individual output is visible and the attrition is not yet.
What is being tested
Whether you can act on a cost that has not shown up in a metric yet.
Attrition is up 6 points in one org. Where do you look first?
What you said
A strong answer
Segment it before explaining it — by tenure, level, manager, and regretted versus not. An org-wide number usually hides one team, one level or one manager, and the intervention for each is different. Exit interviews are the least reliable source here because people optimise for references, so weight what stayers say in the same segments.
What is being tested
Whether you disaggregate before diagnosing.
Your strategy backtests well and loses money live. What is the first suspect?
What you said
A strong answer
Execution assumptions — specifically that the backtest filled at a price that never traded, or ignored half the spread, or assumed a size the book could not absorb. Test it by re-running with fills at the worse side of the touch and a realistic participation cap; a strategy whose edge is smaller than the spread was never an edge. Look-ahead in the data is the second suspect and survivorship is the third.
What is being tested
Whether you know that most backtest alpha is a fill assumption.
A signal has a 70% hit rate. Is it good?
What you said
A strong answer
Unknown, because a hit rate says nothing about expectancy. Seventy percent of small wins against thirty percent of large losses is a losing strategy, and it is the standard shape of one. You need average win, average loss and the distribution — and then a control: what does a random entry with the same holding period and the same exit rule return over the same window.
What is being tested
Whether you ask for expectancy and a control. A hit rate quoted alone is the most common way people fool themselves.
How do you size a position?
What you said
A strong answer
From the distance to your invalidation level and the fraction of capital you are willing to lose on one idea, not from conviction. That makes volatility an input rather than an afterthought, which is why the volatility family of indicators earns its keep when the others do not. Sizing by conviction is how a correct thesis still ends an account, because the path matters and not just the destination.
What is being tested
Whether risk is the input or the leftover.
Market order or limit order?
What you said
A strong answer
It is a trade between certainty of fill and certainty of price, and the right answer depends on whether the cost of not filling exceeds the spread. Liquid instrument, small size, information-driven entry: cross the spread. Illiquid, large relative to the book, or a level-based entry: work a limit and accept misses. Traders who always use one or the other have not priced the other side.
What is being tested
Whether you can name what each order type is actually buying.
What do you do when a position moves against you to your stop?
What you said
A strong answer
Exit at the stop. The decision that mattered was made when the level was set, with a clear head and before money was on it; moving it in the moment is not new analysis, it is the same analysis under duress. The one honest exception is when the invalidation thesis itself has changed on new information — and then you should be able to say what the new information is.
What is being tested
Whether your process survives being wrong. This is the whole job.
Why can the same instrument show different prices at the same instant?
What you said
A strong answer
Because there is no single price — there is a book per venue, and what you see depends on which venue, whether the feed is consolidated, whether it is last-traded or best-bid-offer, and how stale your snapshot is. Add the timestamp question: exchange time, gateway time or your own clock, and how far apart those are. Most reconciliation arguments are really a timestamp definition argument.
What is being tested
Whether you understand market data as plumbing rather than as a number.
Your edge worked for three years and stopped. What happened?
What you said
A strong answer
Ask whether the market changed or whether the edge was never there. The honest tests are out-of-sample performance across regimes, whether the profitable window was one unusual period, and whether the strategy's premise still describes the venue — participants, tick size, latency and fragmentation all change. Assuming decay when the answer is overfitting is how people rebuild the same losing system.
What is being tested
Whether you consider that it may never have worked, which is the more common answer.
The PM hands you a solution rather than a problem. What do you do?
What you said
A strong answer
Work backwards to the problem and the evidence for it, then design against that — and if the handed-down solution is still the best answer, you have lost nothing and gained the ability to defend it. What you cannot do is build it and hope, because the first review question will be why, and “it was what I was given” is not an answer anyone accepts from a designer.
What is being tested
Whether you own the problem definition or just the pixels.
Three teams have forked the button component. What now?
What you said
A strong answer
Find out what each fork needed that the system did not provide, because a fork is a feature request expressed in code. Fix the gap in the system and make adoption cheaper than maintaining a fork; governance without that just moves the forking somewhere less visible. A design system with no forks is either very good or has no users.
What is being tested
Whether you run the system as a product with customers who can defect.
Accessibility work is competing with a launch date. How do you argue it?
What you said
A strong answer
In the specific: which users cannot complete which task, what the legal exposure is in the markets you sell to, and what retrofitting will cost against doing it now. Keyboard navigation, focus order, contrast and labelled controls are cheap during build and expensive afterwards because they are structural. Arguing it as a principle loses to a date; arguing it as compounding cost usually does not.
What is being tested
Whether you can make a quality argument in the currency the decision is being made in.
Latency is up 40% and nothing was deployed. Where do you look?
What you said
A strong answer
Something changed even if you did not change it — data volume, a dependency's behaviour, a cache hit rate falling, a noisy neighbour, a certificate or DNS change, or a gradual resource exhaustion that just crossed a threshold. Start from the four signals on the service and its dependencies, and specifically check saturation, which is the one people skip and where slow degradation hides.
What is being tested
Whether you can debug without a deploy to blame.
Someone proposes autoscaling to fix a capacity problem. When is that wrong?
What you said
A strong answer
When the bottleneck is not the thing that scales — a shared database, a licence limit, a fixed connection pool, a downstream rate limit — in which case autoscaling adds load to the constraint and makes the incident worse. It is also wrong when the scale-up is slower than the traffic spike, which is common, and when the cost of the idle floor was never the problem.
What is being tested
Whether you find the constraint before adding elasticity around it.
Your fleet is at 15% average CPU. Is that a problem?
What you said
A strong answer
It is a question, not a verdict. Average CPU hides the peak you actually provisioned for, and many workloads are bound by memory, I/O or a licence rather than CPU — so the fleet may be correctly sized for a constraint that is not on the chart. The useful move is to find what actually saturates first and what the headroom on that is, then price the risk of reducing it.
What is being tested
Whether you check which resource is binding before resizing on the visible one.
The board asks how productive your engineering org is. What do you say?
What you said
A strong answer
Give outcomes with delivery signals underneath — what shipped, what it moved, and the throughput and stability trend that made it possible. Refuse output proxies like lines of code or story points, and say why: they are gameable in a week and the gaming is worse than the ignorance. The honest version names what you cannot measure yet and what you are doing about it.
What is being tested
Whether you can resist offering a number you know is wrong to an audience that wants one.
Two of your seniors will not work together. What do you do?
What you said
A strong answer
Talk to each separately to find whether it is a disagreement about the work or about each other, because the fixes are different. A technical disagreement gets a decision and a written rationale — someone has to decide. A personal one gets explicit behavioural expectations and a structural change if it does not move. What you cannot do is route around it, because the team is already routing around it and paying for it.
What is being tested
Whether you intervene early. The cost is almost always in what the rest of the team learns is acceptable.
You inherit an org where everything is a priority. First 30 days?
What you said
A strong answer
Find out what is actually being worked on versus what is on the list, then cut publicly and say what you are cutting and why. The list is a symptom — usually of a leadership team that has never had to trade off in the same room. Making one visible, unpopular cut early is what makes the next one possible, and it tells the org that priority means something again.
What is being tested
Whether you can make a cut before you have full context. Waiting for certainty is the failure mode here.
Your site is being measured on cost per head. How do you change that?
What you said
A strong answer
Change what the site owns before you argue about the metric, because the metric follows the scope. Take end-to-end ownership of something with a business outcome attached, deliver it, and then the conversation is about outcomes rather than headcount. Sites that argue for a better metric while owning only execution work do not win that argument, and the reason is structural rather than about talent.
What is being tested
Whether you understand that the measure reflects the mandate, not the other way round.
How to use spoken practice
Every decisive round on this site is spoken, so rehearsing in your head is the wrong medium. Press Answer aloud and talk — your speech is transcribed in the browser and shown back with a word count and a timer, next to the length of the strong answer. Then reveal and compare. Read the answer plays the strong answer back so you can hear the shape of a full response.
Speech recognition runs in the browser and works best in Chrome and Safari; read-back works everywhere. Nothing is uploaded by this site and nothing is stored — your transcript disappears when you close the page.