Infrastructure & Reliability · Reliability, operations & platforms · Chapter 16 of 33
AI in observability, honestly
What works and why, the benchmark that bounds root-cause claims, and the longitudinal number that went the wrong way.
Every vendor in this category now sells an AI story. Some of it is genuinely useful, some is a rebrand, and the evidence base is thinner than the marketing implies in a way that is worth being precise about.
What genuinely works, with the mechanism
| Capability | Why it actually works |
|---|---|
| Log pattern extraction | Deterministic-ish string clustering turns ten million lines into four hundred templates. The most reliably valuable AI in the category and the least marketed, because it is unglamorous |
| Anomaly detection on well-behaved series | Seasonality decomposition is solved statistics. Works on metrics with real daily and weekly structure; produces false positives on anything spiky or deploy-driven |
| Alert correlation and grouping | Four hundred pages become one incident. Works best where the system has a real topology graph to propagate along rather than guessing from co-timing |
| Forecasting | Disk full in N days, quota exhaustion. Linear extrapolation on monotonic resources. Boring, reliable, genuinely saves incidents |
| Natural-language query generation | Real, modest value — query-language friction is the main reason non-specialists do not self-serve. The failure mode is a plausible wrong query, so the generated query must be shown, not hidden |
What is mostly marketing
“Root cause analysis.” The word cause is doing enormous unearned work. What these systems produce is a ranked correlation over a dependency graph. That is genuinely useful triage, and it is not causation — and it systematically cannot find the causes that matter most: a config change, a feature-flag flip, a schema migration, a third party's incident. Anything whose signal is not in the telemetry.
The academic bound is worth knowing. The leading public benchmark for root-cause analysis — 735 failure cases, fifteen methods, across systems up to sixty-four services — scores the best method at about 0.80 on ‘the true cause appears somewhere in the top five candidates’. On synthetically injected, cleanly labelled faults. Production has more services, no ground truth, multiple simultaneous degradations, and causes that are not in the telemetry at all.
“Self-healing.” Restarting a pod on an out-of-memory kill is a health check with better branding. Genuine auto-remediation needs high confidence in the diagnosis and a bounded blast radius, and the first of those is exactly what is not established.
“Reduces time-to-resolution by ninety percent.” Every such figure I could trace is vendor-internal, measured against an undisclosed baseline on an undisclosed dataset. Ninety percent faster than what — a human at three in the morning? a human with the same dashboards? the same human on their second incident of the night? The comparison is unstated and it is the entire content of the claim.
The number that should be set against every vendor claim
There is no independent evaluation of commercial AIOps effectiveness that I could find — no third-party benchmark, no audited time-to-resolution study, no controlled trial. That absence is itself the finding, and it should be stated rather than papered over.
What does exist is a longitudinal practitioner measure, and it went the wrong way. A long-running reliability survey found median reported toil rose from 25% to 30% — the first increase in five years — during the period of maximum AIOps investment, with the report suggesting AI is filling freed capacity rather than reducing workload.
Caveat it properly: the sample is a few hundred, self-selected; correlation is not causation; the adoption confound is unmeasured. But it is a real time series from a consistent methodology, and it is the only thing available to set against a vendor number.
So the useful posture is neither enthusiasm nor dismissal. Ask what the mechanism is. Correlation over a real topology graph has a mechanism. ‘Our AI finds root causes’ does not.
Does AI reduce alert fatigue, or move it?
It cuts alert volume and raises alert stakes, and whether that is net better depends entirely on precision. Grouping four hundred symptoms into one incident is a real win. Three things push the other way:
- Anomaly detection creates a new alert class with no human-authored threshold. Nobody chose it, so nobody feels able to delete it, and it accumulates.
- A grouping failure is worse than the noise it replaced. When correlation wrongly merges two separate incidents you have one page for two problems, and the second is now invisible. A false negative from grouping is strictly worse than ungrouped alerts.
- Trust asymmetry. Once a team learns the summary is sometimes wrong, they re-check it manually — and now they are doing both jobs.
And the measurement point that matters: the honest measure of an AIOps deployment is not ‘alerts reduced by X percent’, which is trivially gameable by suppression. It is precision — what fraction of pages were worth waking for — and recall — what fraction of real incidents produced a page at all. Survey work suggests a large majority of organisations have had an incident where no alert fired. Recall is the number nobody measures, and every ‘we cut alerts by eighty percent’ story is a precision story that says nothing about it.
Check yourself