Platform & Developer Experience
Observability Engineer
Separating out into its own discipline right now — the platform family's equivalent of evaluation in the AI family.
What the job actually is
You make systems explicable: metrics, logs, traces, and the increasingly expensive question of which of them to keep. The reason this is becoming a standalone role is cost — telemetry volume grows faster than the systems producing it, and someone has to own cardinality, sampling strategy and the bill.
The parallel with evaluation is exact: observability is named in 42% of postings and titled in far fewer, which means most people assessed on it have never studied it deliberately.
Titles this hides behind
Prepare like this
- Understand cardinality properly. Why adding a user ID label to a Prometheus metric can cost six figures, and what you do instead.
- Know sampling strategies — head, tail, and why tail sampling is both what you want and operationally painful.
- Instrument something real with OpenTelemetry end to end, and be able to explain the collector's role.
- Have a cost story. Telemetry spend reduced without losing the signal that mattered — this is the interview's favourite territory.
Be ready for
- "Your observability bill is larger than your compute bill. What do you cut first?" Increasingly a real situation, and the reason the role exists.
- "A request is slow but every service reports healthy. How do you find it?"
- "Metrics, logs or traces — you can keep two. Choose and defend it."
Compensation
A day that is actually representative
| Time | What you are actually doing |
|---|---|
| 09:00 | Cardinality report — which metric grew and why. |
| 11:00 | Instrumenting a service properly with OpenTelemetry. |
| 13:30 | A slow request nobody can find, which is the job. |
| 15:00 | Sampling policy, and what you lose. |
| 16:30 | The telemetry bill, and which queries justify their cost. |
Real cases worth being able to discuss
Across the industry, observability spend has repeatedly grown to rival compute, driven almost entirely by high-cardinality labels added without review.
Use it to say: “It is why I treat a new metric definition like a schema change — reviewed, with cardinality estimated up front.”
A three-week plan
Week 1 — instrument
Days 1–3: End-to-end OpenTelemetry on a real service, including the collector.
Days 4–5: Deliberately create a cardinality explosion and watch the series count.
Week 2 — sample
Days 1–3: Implement tail sampling and articulate what you keep and lose.
Days 4–5: Produce a cost-per-useful-query estimate.
Week 3 — rehearse
Days 1–3: The observability-bill answer, out loud.
Days 4–5: The healthy-services-but-slow-request debugging narrative.
The book for this field
Infrastructure & Reliability
Reliability and error budgets, capacity arithmetic, networks, storage and the restore you have not tested, the physical layer, a debugging method that transfers, running production, and an honest look at Kubernetes. Plus a Middleware section — web and application servers, messaging (IBM MQ, Kafka) and managed file transfer, from basics to licensing and cost.
Also read for this role
SDLC, DevOps & Developer Experience
How software actually gets from a developer's machine to production: deployment strategies and the version-skew problem underneath them, infrastructure as code and the state file that runs everything, pipeline design that people do not route around, and the software supply chain link by link.
Cross-cutting
Skills every archetype tests
These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.
Practice the 35 Platform questions → · Back to Platform & Developer Experience →