Data Platform & Analytics · Chapter 3 of 16
Data contracts, and who gets paged
The structural fix for the defining problem of the field: your pipeline breaks because of a change you did not make.
The characteristic experience of data engineering is being paged for something a service team shipped. They renamed a column, changed a unit, started sending nulls — entirely reasonably, from their point of view, because nobody told them anyone downstream depended on it.
What a contract actually contains
- Schema — fields, types, nullability, and which fields are guaranteed.
- Semantics — what the field means, its unit and its currency. More breakages come from a silent unit change than from a type change.
- Guarantees — freshness, completeness, and whether it is append-only.
- Ownership — a named producing team, because a contract with no owner is documentation.
- A version and a deprecation policy, so change is possible rather than forbidden.
Getting them adopted, which is the hard part
Contracts fail politically rather than technically. Producers experience them as a constraint imposed by a team they do not report to. What works: start with the two or three sources that have actually caused incidents rather than mandating coverage; make the check trivially cheap to add; and bring the incident record — “this change cost us eleven hours and the finance report was wrong for two days” is far more persuasive than a governance principle.
Making a contract that actually holds
A data contract is only worth the mechanism behind it. Written down and unenforced, it is documentation that ages badly — and it ages worse than ordinary documentation because people trust it.
- Put it at the boundary you do not control. Between an application team who own a source database and a data team who consume it, a contract solves a real problem: the silent Friday schema change. Inside one team that already reviews each other's changes, it is ceremony over a review you were already doing.
- Specify the semantics, not just the schema. Types are the easy half. The contract needs to say what the field means, its units and currency, its cardinality, whether nulls are permitted and what a null signifies, and what the grain is.
- Version it, and define what a breaking change is — adding an optional field is not; changing a type, tightening nullability or altering a meaning is.
- State freshness as a commitment, because most consumer breakage is lateness rather than wrongness.
- Name an owner who can be asked. A contract without a name on it has no escalation path, and the first time it is violated you will discover there is nobody to talk to.
The question that reveals whether a contract is real
“What happens, mechanically, when a producer violates it?”
If the answer is ‘an alert fires’, that is monitoring with better documentation. Useful, and not a contract.
If the answer is ‘their deployment fails’ or ‘the message is rejected at the registry’, that is a contract.
This matters because most tooling in the category does not enforce. It authors, validates the document against a specification, and generates artefacts. Blocking a violating write requires sitting in the write path, which most platforms deliberately avoid.
So enforcement in practice is assembled from three things you own: a continuous-integration check on the producer's repository that fails the build on an incompatible change; runtime validation at the ingestion boundary that quarantines rather than propagates; and schema-registry compatibility rules if the data arrives over a streaming platform.
None of that is exotic. It is just work that the word ‘contract’ makes people think they have already done.
Check yourself