Infrastructure & Reliability · Reliability, operations & platforms · Chapter 12 of 33
Observability: what to instrument, and the bill you will get
Three signals, one cardinality trap that has bankrupted monitoring budgets, and the difference between a dashboard and an answer.
Monitoring tells you whether the things you predicted would break have broken. Observability is the ability to answer questions you did not think of in advance. Most organisations have a lot of the first and believe it is the second, and the gap shows up during an incident when nobody can answer ‘which customers were affected?’.
The three signals, and choosing between them
- Metrics are aggregates over time — a counter, a gauge, a histogram. Cheap, always on, ideal for alerting and for trend. They cannot tell you about an individual request, by construction.
- Logs are discrete events with detail. They answer ‘what exactly happened’ and they get expensive fast at volume. Structured logs — JSON with consistent field names — are worth the migration, because grepping unstructured text does not scale past a few services and cannot be aggregated.
- Traces follow one request across service boundaries, showing where the time went. This is the signal that answers ‘why is this endpoint slow’, and it is the one most organisations have least of.
- Profiles are increasingly the fourth: continuous, low-overhead CPU and memory profiling in production. This is where the field has moved and it is worth knowing about.
The cardinality rule, stated once so it sticks
Every unique combination of label values creates a separate time series. They multiply.
Adding user_id as a label to a metric with 40,000 existing series and 200,000 users produces eight billion series. Your time-series database will fall over and your bill will be extraordinary.
A label's value set must be small and bounded. Service, endpoint pattern (not the raw path with IDs in it), status class, region — fine. User ID, request ID, email address, full URL, raw error message — never. Those belong in logs and traces, which are designed for high-cardinality detail.
This single mistake is the most common cause of both monitoring outages and shocking observability invoices, and knowing it unprompted is a reliable signal of production experience.
What to actually instrument
Two complementary checklists, and knowing when to use which is the skill.
| Method | For | The three things |
|---|---|---|
| RED | Request-driven services | Rate — requests per second · Errors — failures per second · Duration — the latency distribution |
| USE | Resources | Utilisation · Saturation · Errors — as in the Linux performance chapter |
A practical addition that carries a lot of weight: instrument the business event, not only the technical one. Orders placed per minute detects a broken checkout faster and more reliably than any infrastructure metric, because it measures the thing you actually care about and it is insensitive to which component failed.
Histograms, percentiles, and the arithmetic that is wrong
You cannot average percentiles. The mean of the p99 across ten hosts is not the fleet p99, and it is not an approximation of it either — it can be wildly off in either direction. Percentiles must be computed from the merged distribution, which is why histogram metrics store buckets rather than computed quantiles.
Bucket boundaries decide your resolution. If your histogram has a bucket edge at 1 second and your p99 sits at 1.4 seconds, the estimate is interpolated between 1 and the next edge and can be substantially wrong. Choose boundaries around the values you actually care about — typically your SLO threshold — rather than accepting the library defaults.
Average latency is close to useless and should rarely appear on a dashboard. If 1% of requests take 10 seconds and 99% take 10 ms, the mean is 110 ms and nobody experiences it.
And measure where the user is. Server-side latency excludes DNS, TCP, TLS, the network and the client's own rendering. A request that never arrived is a failure to the user and is invisible in your server metrics — which is exactly the class of outage that gets reported on social media before it appears on your dashboard.
Sampling, because you cannot trace everything
- Head sampling decides at the start of the request, before you know anything. Cheap, and it discards the interesting traces at the same rate as the boring ones.
- Tail sampling buffers the whole trace and decides after seeing it — keep all errors, all slow requests, and 1% of the rest. Far more useful and far more expensive, because something must hold the spans until the decision.
- The pragmatic default is head sampling at a low rate with a forced-sample header so you can trace a specific request on demand, plus complete capture of errors. That gets most of the value at a fraction of the cost.
- Whatever you choose, propagate context. A trace that stops at a service boundary because someone did not forward the headers is worse than no trace, because it points confidently at the wrong component.
Alerting: the discipline, not the tool
The failure mode of alerting is not missing alerts — it is too many, because an on-call engineer who receives fifteen pages a night stops reading them, and then the one that mattered is missed. Alert fatigue is a measurable, well-documented cause of severe incidents.
- Alert on symptoms, not causes. ‘Error rate above 2% for five minutes’ is worth waking someone. ‘CPU above 80%’ is not — a busy machine serving traffic perfectly is not an incident, and if high CPU never causes a symptom you have paged someone for nothing.
- Every page needs a runbook and a plausible action. If the recipient cannot do anything, it is not a page; it is a notification, and it belongs somewhere else.
- Burn-rate alerting on an error budget is the mature form: alert when you are consuming the budget fast enough to exhaust it before the window ends. Fast burn pages; slow burn creates a ticket. This automatically scales urgency to actual impact, which fixed thresholds cannot do.
- Review your pages monthly and delete the ones nobody acted on. This is the single highest-value observability practice and almost nobody does it.
Check yourself