Lakshya

Handbook · Chapter 4

Latency and cost arithmetic

The one piece of maths that is genuinely counterintuitive — and it comes up in every agent interview.

Cost and latency questions look like trivia and are not. They are the fastest way for an interviewer to find out whether you have operated a system or only built one. The single non-obvious result is what happens to tail latency when you chain calls — which is exactly what every agent does.

ONE AGENT TURN — FOUR SEQUENTIAL CALLS retrievep50 40msp99 200msplanp50 40msp99 200mstool callp50 40msp99 200mssummarisep50 40msp99 200ms WHAT THE USER ACTUALLY WAITS 160ms — every call at its median 320ms — three medians plus one tail The chain has no p99 of its own — it inherits four chances to tail. P(at least one call in its tail) = 1 − 0.99⁴ = 3.9% of turns, not 1%. So the chain’s p96 already looks like a single call’s p99. Adding a fifth call makes it 4.9%.
A chain does not have a p99; it has four chances to hit one. If each call has a 1% tail, the probability that a turn hits at least one tail is 1 − 0.99⁴ = 3.9%. So roughly one turn in twenty-five behaves like a worst case, and your users experience the tail four times more often than your per-call dashboard suggests.

What follows from that

  • Per-call dashboards understate user-visible slowness. Instrument the turn, not just the call. This is the answer to “every service reports healthy but requests are slow”.
  • Adding a step is more expensive than it looks. A fifth call moves you from 3.9% to 4.9% of turns in the tail — and it adds its median to every single turn.
  • Parallelise what you can, because parallel calls take the max rather than the sum, and the tail probability is the same either way.
  • Timeouts are a design decision, not a config value. A timeout below the downstream p99 converts slow requests into failed ones, which may be what you want — but only if you say so deliberately.

Cost: the denominator is the whole question

Interviewers ask about cost per request. The better answer volunteers cost per resolved task, because that is the number a business runs on and because it exposes retries. An agent that costs $0.04 a call and needs three attempts costs $0.12 to succeed — and if it fails 10% of the time and escalates to a human, the true unit cost includes that human.

MetricWhat it hidesAsk instead
cost per API callretries, failed turnscost per resolved task
tokens per requestthe context you resend every turntokens per conversation
p50 latencythe chained tail abovep99 measured at the turn
GPU utilisationidle time between batchestokens per second per dollar
telemetry volumecardinality explosionscost per useful query

The observability version of the same problem

Platform candidates get this as: “your observability bill is larger than your compute bill — what do you cut?” The mechanism is cardinality. Adding one high-cardinality label — a user ID, a request ID, a full URL — to a metric multiplies the stored time series by the number of distinct values, and that is how a single well-meaning line of code produces a six-figure invoice.

The answer structure: move high-cardinality data from metrics to traces, sample traces at the tail so you keep the slow ones, and keep aggregate metrics low-cardinality. Then say which queries you would lose, because a cut with no cost is a cut nobody believes.

The drill

  • Take an agent you have built. Instrument each step and the whole turn. Compare the turn's p99 to the sum of the step medians — the gap is the thing this chapter is about.
  • Compute your cost per resolved task including retries. Most people are surprised by a factor of two to three.
  • Then halve one of them and be able to say what you traded.
← Design under constraintDiscovery before solution →