Data Platform & Analytics · Chapter 15 of 16
Data quality, observability and contracts
What monitoring genuinely detects, the class of wrongness it structurally cannot, and why most contract tooling does not enforce contracts.
“Data quality” covers three different things that are constantly conflated: testing (assertions you write), observability (statistical monitoring you configure), and contracts (agreements between producer and consumer). Knowing which one solves which problem is most of the skill.
What monitoring can and cannot see
Every observability platform detects the same four things well: freshness, volume, schema and distribution. Those are genuinely valuable and genuinely automatable — they catch the pipeline that silently stopped, the upstream column that changed type, the day the null rate tripled.
The class they structurally cannot detect is semantically wrong data at stable volume.
A join grain that changed from one-to-one to one-to-many. A new region writing local currency into a column everyone assumes is dollars. A filter that was dropped in a refactor, quietly returning test accounts to a production aggregate.
In all three cases the row count is normal, the schema is unchanged and the distribution barely moves. Nothing statistical fires. The number is simply wrong, and somebody makes a decision on it.
The only defence is a human assertion about meaning — a uniqueness test at the expected grain, a contract that pins the currency, a reconciliation to a total you can verify independently.
So observability and testing are complements, not substitutes. Teams that buy a platform and let their test suite rot have moved their coverage backwards while feeling more covered.
And the operational failure of observability is not missed detection — it is alert fatigue. Turn anomaly detection on for every column of every table and you generate more alerts than any human reads. A channel nobody reads is worse than no channel, because it converts a real incident into a notification somebody scrolled past. Instrument the tables that feed decisions, leave the rest deliberately unmonitored, and make that a stated choice.
The tooling, and a licence story that repeats
The open-source quality tools split along a familiar line. One long-standing framework, built around declarative expectations with data docs and profiling, is powerful and has a steep learning curve. Another uses a compact check language in YAML and is far quicker to adopt. A third is simply the tests built into your transformation framework — not-null, unique, accepted values, referential integrity — which cost nothing extra and cover a surprising share of real defects.
Start with the transformation framework's built-in tests. They are free, they live beside the model, they run in the same job, and most teams never exhaust their value. Add a dedicated framework when you need assertions the framework cannot express, and a commercial platform when you need cross-system lineage and incident workflow that no amount of in-pipeline testing gives you.
The corporate history is instructive: the long-standing framework's company sold its commercial cloud product to an analytics firm — and that product became publicly unavailable — while stewardship of the open-source core passed to the ingestion vendor from the previous chapter. Meanwhile the compact-check-language tool relicensed its core away from a permissive licence.
The lesson is not about any one product. It is that the open-source core and the commercial product around it can be separated and sold to different owners — so “is it open source?” is the wrong question. The right ones are: who holds the trademark, who merges the pull requests, and what happens to my deployment if the commercial arm is sold tomorrow?
Data contracts, and the uncomfortable finding
A data contract is an explicit, versioned agreement between the team producing a dataset and the teams consuming it: schema, semantics, quality guarantees, freshness expectations, and a change process.
The specification landscape consolidated usefully. Two competing open specifications merged — one was formally deprecated in favour of the other, which now sits under a neutral foundation and reached a mature version in late 2025. That is a genuine win: one specification rather than two is exactly what a standard is for.
Writing contracts is solved. Enforcing them is not.
The single most useful finding in this whole area: most data contract tooling does not actually enforce contracts.
What the tools reliably do is author them, validate the document against the specification, and generate artefacts — schemas, documentation, test stubs.
What almost none of them do is block a violating write at the boundary. Enforcement in practice is a continuous-integration check on the producer's repository, plus runtime validation you wrote yourself, plus schema-registry compatibility rules if the data arrives over a streaming platform.
So when someone says ‘we have data contracts’, the question that finds the truth is “what happens, mechanically, when a producer violates one?” If the answer is ‘an alert fires’, that is monitoring with better documentation. If the answer is ‘the deployment fails’ or ‘the message is rejected’, that is a contract.
This is not a criticism of the tools — blocking a write requires sitting in the write path, which is an architectural commitment most platforms deliberately avoid. It is a criticism of how the category is described.
Where contracts genuinely pay: at organisational boundaries, where an application team owns a source system and a data team consumes it, and the failure mode is a silent upstream schema change on a Friday. Inside one team that already reviews each other's pull requests, a contract is usually ceremony over a review you were already doing.
Check yourself