Infrastructure & Reliability · Reliability, operations & platforms · Chapter 15 of 33
The observability bill, and why cardinality is the cause
A new label multiplies rather than adds, sampling loses either the rare or your budget, and every migration case study has the same fatal confound.
Observability cost is not a procurement problem that occasionally becomes an engineering one. It is an engineering problem that arrives as an invoice, and the mechanism is nearly always the same.
Cardinality: the multiplier nobody budgets for
A timeseries is identified by its metric name plus the full set of label key-value pairs. Adding a label does not add one series — it multiplies the series count by that label's distinct-value count. Datadog's own documentation is explicit: a custom metric is uniquely identified by a combination of metric name and tag values.
So one metric with three labels of ten, twenty and fifty values is not one metric. It is ten thousand billable series.
| Label you added | Distinct values | Effect on that metric |
|---|---|---|
status_code | ~40 | ×40 — fine, and bounded |
pod_name | one per pod per deploy — unbounded over time | Grows forever. Old series persist for the whole retention window |
customer_id | your customer count | ×N, and it grows every time sales succeed |
user_id | your monthly actives | Catastrophic |
trace_id / request_id | your request count | Immediately fatal |
url with path parameters | distinct orders | Fatal, and it looks completely innocuous in review |
error_message (raw string) | distinct stack traces | Fatal — and the single most common accident |
Note that the guidance from the major projects gives no numeric threshold. The rule is bounded versus unbounded, not ‘under N’. A five-hundred-value label that will never grow is fine. A twenty-value label that grows with traffic is a time bomb.
Log platforms have the identical problem in a different shape. A label-indexing log store creates a stream per unique label combination, so high cardinality builds a huge index and flushes thousands of tiny chunks. Their documentation says the same thing: do not label on pod, request identifier, user, trace or address.
And a specific trap worth knowing: on at least one platform a histogram data point bills as eight series. A twenty-bucket histogram across three services and five endpoints is not three hundred billing units, it is nearer two thousand four hundred.
Why nobody budgets for it, structurally: the person who creates the cost is not the person who receives the bill; the unit is invisible in code, because adding a tag looks like one metric; and it is not linear, so every capacity model built on ‘gigabytes per host times hosts’ misses it entirely.
The control that works is upstream: a cardinality budget enforced in continuous integration, failing the build when a new label's value set is unbounded. Every downstream remedy the vendors sell is, structurally, ‘let us aggregate away the dimension you should not have added’.
Sampling: head loses the rare, tail loses the weekend
Head sampling decides early, usually on a trace identifier, before the trace exists. It is trivial to configure, stateless, and reduces cost at the source. And the specification says plainly what it costs: you cannot ensure that all traces with an error are sampled with head sampling alone. At one percent, a bug affecting fifty users a day is invisible — you will see zero or one example and no pattern. Head sampling is a rate limiter, not an observability strategy.
Tail sampling buffers complete traces and decides on latency, errors or attributes — keep every error and slow trace, keep a fraction of a percent of the boring ones. That is what you actually want, and here is what it costs:
- All spans of a trace must reach the same collector instance. So you need a two-tier deployment with trace-identifier-aware load balancing. Scale your collectors horizontally behind an ordinary load balancer and tail sampling silently makes wrong decisions on fragmented traces. This is the trap that catches everybody.
- It is a stateful, capacity-planned service. A bounded buffer, and overflow drops traces before evaluation — visible only in a metric almost nobody alerts on.
- A decision wait window means a trace longer than the window is decided on incomplete data, and increasing it increases memory.
- And metrics derived from sampled traces are wrong unless the sampler propagates its rate — a subtlety the standard probabilistic sampler handles and many home-rolled ones do not.
The pragmatic middle most mature teams land on: head-sample generously to control raw volume, tail-sample in a gateway tier for errors and latency outliers, and keep a few explicitly chosen critical paths unsampled.
Every migration case study has the same fatal confound
The genre is familiar: “we moved off vendor X to Y and cut costs 40–60%.”
Every named case study I could find is published by the vendor being migrated to. No independent, third-party-audited observability migration study appears to exist.
And the confound is fatal: a migration always bundles a telemetry-hygiene exercise — dropping data nobody queries, fixing the labels that exploded, deleting dashboards nobody opens. The hygiene is doing most of the work, and a team that ran the same audit and stayed put would capture much of the same saving.
The savings cluster at 40–60% because that is roughly what any competent re-architecture yields, not because of the new vendor's technology.
The best-documented case ends the other way: a company with a very large annual bill built an open-source replacement, ran it double-write — and stayed with the commercial vendor after a steep discount. The credible threat of migration was worth more than the migration. Use these stories as evidence about negotiating leverage, not about open source winning.
And if you are considering self-hosting: run the arithmetic honestly. The comparison is not licence versus zero. It is vendor bill versus infrastructure plus a full-time engineer plus the incidents you will have while learning it. The crossover is much further out than people assume.
Check yourself