Infrastructure & Reliability · Reliability, operations & platforms · Chapter 13 of 33
The observability tool landscape
What each product is genuinely for, what it is genuinely bad at, and the licence and pricing history that explains why the market looks like this.
The previous chapter covered what to instrument. This one covers what you buy, because “we use Datadog” and “we use Prometheus and Grafana” describe different engineering lives, and the choice is usually made badly — on a feature comparison, when the things that actually decide it are team skills, cardinality and who owns the platform.
The products, honestly
| Genuinely good at | Genuinely bad at | |
|---|---|---|
| Splunk (Cisco, closed Mar 2024) | Search everything, schema unknown, retain for years, prove it to an auditor. Security is its centre of gravity, not APM | Two products sharing a brand — the log platform and the observability platform have different engines, query languages and UIs. Unit cost at volume |
| Datadog | Breadth, integrations, and the best incident-time UX in the category. Fastest time to a useful dashboard | Cost predictability. Per-host plus per-custom-metric plus per-GB, and the custom-metric meter is driven by code developers change without an approval gate |
| Dynatrace | Large Java/.NET estates where nobody will hand-instrument. Its agent auto-discovers topology, and correlation runs over that graph rather than over timestamps | Small teams and greenfield. The automation is the lock-in — the value comes from the agent being everywhere, and its data model is theirs |
| New Relic | The most generous free tier in the category — 100 GB a month and unlimited read-only users, which removes the 'we can't afford to let everyone look' problem | Ingest-metered cost risk, same shape as everyone else's. Taken private in Nov 2023 |
| Elastic | Log search at scale on infrastructure you control; full-text and vector search in one engine | Operational burden — cluster, shard and Java Virtual Machine (JVM) tuning is a real job. Metrics are weaker than logs. And it bills on post-enrichment size, which can exceed your raw log volume |
| Grafana stack | Being the neutral pane of glass over datasources you already have — still the most common reason it is in a stack. Lowest unit cost at volume | Self-hosted, it is a distributed-systems project, not an install. Each component is a microservice fleet with object storage, rings and compactors |
| Prometheus | The data model and exposition format the whole industry adopted. Excellent Kubernetes service discovery | Long-term storage (single node, no replication — hence Thanos, Cortex, Mimir), high cardinality, and anything event-shaped |
| Honeycomb | Unknown-unknowns. Wide events with no write-time aggregation, so cardinality is free at query time — group by customer or build ID after the fact | Infrastructure monitoring breadth. And it demands real instrumentation discipline: point it at thin auto-instrumented spans and you get a nicer trace viewer |
| Chronosphere | Cost control as the product, for very large Prometheus estates where cardinality is the binding constraint | Being your only tool. It is deliberately not a full-breadth suite |
Two licence histories, and why they ended differently
Same problem, same year, opposite choices — and the outcomes followed the choice. This is the cleanest natural experiment in open-source licensing.
Elastic moved off Apache 2.0 in early 2021 to a dual licence, neither of which is approved by the Open Source Initiative (OSI), the body that defines the term. It was forked into OpenSearch within about two months, and that fork now sits under a neutral foundation with hyperscaler backing.
Grafana moved to AGPLv3 a couple of months later — copyleft, but OSI-approved and FSF-free, so users kept the freedoms they had. It was not forked.
And the correction almost everyone gets wrong: Elastic added AGPLv3 as a source-code option in 2024, and the headlines read ‘open source again’. Elastic's own licensing FAQ still says the default distribution remains under the Elastic License — which is not OSI-approved. The source has an open option; the binary most people download does not. Read the FAQ, not the blog title.
The transferable lesson: a licence that removes a user freedom invites a fork; a copyleft licence that preserves user freedom while blocking the hyperscaler resale does not. That distinction predicted both outcomes.
The pricing models, and what each punishes
| Model | Punishes | Rewards |
|---|---|---|
| Per host | Many small hosts, autoscaling, short-lived nodes — and on Dynatrace, memory-heavy hosts, because billing is per memory GiB-hour rather than per core | Few large hosts, stable fleets |
| Per GB ingested | Verbose logging, debug level in production, auto-instrumentation — anything a developer changes without review | Terse, structured, sampled telemetry |
| Per custom metric | Cardinality. Full stop. A new label multiplies, it does not add | Bounded, deliberate label sets |
| Per user | Large orgs wanting broad access | Small teams, or read-only viewers |
| Per query / scanned GB | The incident where you actually need to look at everything. The correct engineering behaviour is the expensive behaviour | Store-a-lot-query-rarely archives |
| Per event | Chatty services and deep span trees | Wide events over many events |
The pay-per-query model deserves a specific warning, because it is growing and the flaw is structural: it is excellent right until you have a real incident, at which point querying aggressively across everything is both the right thing to do and the thing that costs money. Any pay-per-query contract needs an explicit incident-mode budget, or engineers will hesitate at exactly the wrong moment.
The most-cited cost story in the industry, and how it actually ended
A large crypto exchange's monitoring bill reached roughly $65 million for a single year. The figure was derived by an analyst from the vendor's own earnings-call maths, later confirmed by the vendor's CFO describing a large upfront bill from a crypto customer, and then confirmed as to identity by engineers who were there.
They built a replacement on open-source components and ran it double-write.
And then they stayed on the commercial vendor, after a large discount.
That ending is the lesson, and it is almost never the one people quote. The credible threat of migration was worth more than the migration — because once the price moved, the remaining delta no longer covered the migration cost and the loss of developer experience.
Read every “we moved off X and saved 40%” case study with the same scepticism. Every one I could find is published by the vendor being migrated to. And there is a fatal confound: a migration always bundles a telemetry-hygiene exercise — dropping data nobody queries — and the hygiene is doing most of the work. A team that ran the same audit and stayed put would capture much of the same saving.
One more caution about vendor figures, offered without naming the company because the point is general. One cost-optimisation vendor's own website gives three different numbers — 60%, 84% and 89% — for ‘the average’ data-volume reduction its customers achieve, on different pages, with no methodology, cohort or time window on any of them. A range that wide for a single average means the pages are measuring different things. Quote such numbers as what the vendor claims, never as fact.
Check yourself