Artificial Intelligence
AI Platform & Inference Infrastructure Engineer
Where the GPUs, the throughput and the cost-per-token live. Fewer postings, less competition per posting.
What the job actually is
Making training and inference fast and affordable at scale. Cluster scheduling, distributed training that survives node failure, serving stacks, batching strategy, quantisation, and the unglamorous arithmetic of how many tokens per second you get per dollar. This is the archetype whose skills age best — the specific models change yearly, the systems constraints do not.
Titles this hides behind
Who is hiring it
Baseten, Modal, Together, Fireworks, Databricks, Snowflake, plus the infrastructure organisation inside every frontier lab. In India, Sarvam is hiring both training infrastructure and inference performance, and Glance is hiring on-device inference — which is a distinctly Indian specialism driven by a mobile-first market.
How to recognise it in a JD
Look for throughput, latency, quantisation,
KV cache, vLLM, Triton, CUDA,
FSDP, multi-node, cost per token. These terms
are rare corpus-wide — CUDA 4%, distributed training 6% — which is precisely why they
are a clean signal when present.
The loop
Prepare like this
- Do the memory arithmetic until it is instant — parameters, optimiser state, activations, and KV cache growth as a function of batch and sequence length. This gets asked directly and answered badly.
- Know continuous batching, paged attention, speculative decoding, and tensor versus pipeline parallelism well enough to explain the trade-off, not just the name.
- Bring one measured win. A latency or cost-per-token improvement with a before, an after, and the profiling that found it.
- Serve a real model yourself on rented GPUs and hit an actual throughput target. The gap between reading and doing shows immediately here.
Be ready for
- "Estimate the GPU memory to serve a 70B model at batch 32, 8k context." Arithmetic, out loud, with your assumptions stated.
- "p50 latency is fine, p99 is terrible. Where do you look?" Queueing, batching, preemption, and cold starts.
- "Cut inference cost 40% without hurting quality. What do you try, in what order?"
Red flags in the posting
"AI Platform" that is actually Kubernetes. Some postings under this title are general platform engineering with no model-specific work at all. Ask in the screen whether the team owns the serving stack or merely the cluster it runs on.
Compensation
A day that is actually representative
| Time | What you are actually doing |
|---|---|
| 09:00 | Queue depth. Jobs waiting, GPUs idle, and the scheduling policy that is producing both at once. |
| 10:30 | Serving stack. A model update changed memory profile and the batch size that fitted yesterday does not fit today. |
| 13:00 | Profiling. Where the time actually goes in the inference path, which is rarely where people assume. |
| 15:00 | Cost. Tokens per second per dollar, and whether quantisation would buy more than a bigger instance. |
| 16:30 | An incident: p99 spiked and the cause was cold starts on a scaled-up replica set. |
Capacity constraints made utilisation an economic emergency rather than an engineering nicety; organisations discovered how much of their fleet was idle.
Use it to say: “It is why I report tokens per second per dollar rather than raw latency — the business is buying throughput per unit of scarce capacity.”
Week 1 — arithmetic
Days 1–2: Memory decomposition until instant: weights, optimiser state, activations, KV cache.
Days 3–5: Serve a real model on rented GPUs and hit a stated throughput target.
Week 2 — optimisation
Days 1–3: Continuous batching, paged attention, quantisation — measure each, do not assume.
Days 4–5: Produce a before-and-after on latency or cost per token, with the profiling that found it.
Week 3 — rehearse
Days 1–2: The 70B memory estimate, out loud, including the KV cache term.
Days 3–4: The p99-versus-p50 answer and the 40%-cost-reduction sequencing.
Day 5: Five stories, each with a measured number.
The book for this field
LLM Systems Engineering
Seven chapters on the engineering underneath applied AI: the roofline and KV-cache arithmetic that explain every serving decision, quantisation with the evidence, serving stacks and the prompt-cache traps that cost real money, retrieval internals and whether long context replaced RAG, agents at the token level, evaluation with actual statistics, and what won in post-training.
Cross-cutting
Skills every archetype tests
These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.
Practice the 22 AI questions → · Back to Artificial Intelligence →