Data Platform & Analytics · Chapter 4 of 16
Data quality that is not a dashboard
Most quality programmes produce alerts nobody actions. This is about the small number of tests that earn their place.
The failure mode is familiar: hundreds of tests, thousands of alerts, and a team that has learned to ignore them — which means a real failure is now invisible inside the noise, exactly as with a badly tuned SIEM — a Security Information and Event Management system, which centralises logs from everywhere and alerts on them.
Four checks worth having on everything important
| Check | Catches | Why it earns its place |
|---|---|---|
| Freshness | The pipeline did not run | The single highest-value test. Most incidents are absence, not corruption |
| Volume | Row count outside the expected band | Catches partial loads, which are worse than failures because they look successful |
| Uniqueness on the key | Fan-out from an upstream change | Prevents silent double-counting, the hardest bug to notice |
| Null rate on required fields | A producer change | Usually the first symptom of a broken contract |
Almost everything beyond those four is either a business rule that belongs in the model or a test somebody added after an incident and nobody has revisited since.
The metric to report, and the one to avoid
Report: the proportion of incidents found by a check rather than by a human, and the median time from breakage to detection. Those tell you whether the programme works.
Avoid: number of tests, or test coverage. Both reward adding tests, which is the behaviour that produced the noise. A team with 900 tests and a 40% alert-to-action rate is worse off than one with 60 tests and a 90% rate.
Lineage is what makes an incident survivable
When something breaks at 07:00, the question is not what broke — it is what depends on it, so you know who to tell before they find out themselves. Column-level lineage turns a half-day of investigation into a query, and it is also the thing that lets you delete anything with confidence.
Where to put the checks
Quality checks are cheap to write and easy to put in the wrong place. The rule that survives: test at the boundary you do not control, and at the grain you promise.
| Place | What belongs there | What does not |
|---|---|---|
| On ingestion, at the source boundary | Schema conformance, required fields, referential sanity, row-count plausibility | Business logic — you do not yet know the business meaning of raw data |
| On the model you publish | Grain uniqueness, not-null on keys, accepted values, referential integrity to dimensions | Anything that duplicates an ingestion check — you will maintain it twice |
| On the metric | Reconciliation against an independently-known total | Row-level assertions; the metric is an aggregate |
| In monitoring, not tests | Freshness, volume bands, distribution drift | Anything with a definite right answer — that is a test, and it should block |
The distinction in the last row is the one worth internalising. If there is a correct answer, assert it and fail the build. If there is only a normal range, monitor it and alert. Teams that alert on things with definite answers get alert fatigue; teams that test things with only a normal range get a red pipeline every time the business has a good day.
The check that catches the most and costs the least
A uniqueness test on the grain key of every published table. If it is one row per order, assert order ID is unique.
It is one line. It has no false positives — the grain either holds or it does not. And it catches the entire class of silent fan-out from a join to a one-to-many table, which is the most expensive quiet failure in analytics because the numbers stay plausible.
No statistical monitor can catch that class, because after a fan-out every row is a real row, the schema is unchanged and the distribution barely moves. Volume rises, but by an amount that looks like growth.
Second on the list, and nearly as cheap: reconciliation to an independently-known total. If finance has a revenue number produced by a different system, compare to it on a schedule. It is the only check that validates meaning rather than shape, and it is the check that finds the errors everything else was structurally unable to see.
Check yourself