Lakshya

Data Platform & Analytics · Chapter 3 of 8

Data contracts, and who gets paged

The structural fix for the defining problem of the field: your pipeline breaks because of a change you did not make.

3 min read1 diagramAll 8 chapters

The characteristic experience of data engineering is being paged for something a service team shipped. They renamed a column, changed a unit, started sending nulls — entirely reasonably, from their point of view, because nobody told them anyone downstream depended on it.

A PRODUCER RENAMES A COLUMN without a contract service team ships pipeline exec dashboard: blankfinance report: wrongML feature: silently null discovered days later, by a person, not a test with a contract service team contract check in THEIR pipeline build fails, before merge nothing downstream breaks The move that matters: the check runs in the producer's pipeline, not yours. Otherwise you are still the one paged.
Without a contract the breakage is discovered downstream, by a person, days later. With one, the check runs in the producer's own pipeline and the change fails before it merges. The move that matters is whose build goes red — a schema test in your pipeline still leaves you as the one paged.

What a contract actually contains

  • Schema — fields, types, nullability, and which fields are guaranteed.
  • Semantics — what the field means, its unit and its currency. More breakages come from a silent unit change than from a type change.
  • Guarantees — freshness, completeness, and whether it is append-only.
  • Ownership — a named producing team, because a contract with no owner is documentation.
  • A version and a deprecation policy, so change is possible rather than forbidden.

Getting them adopted, which is the hard part

Contracts fail politically rather than technically. Producers experience them as a constraint imposed by a team they do not report to. What works: start with the two or three sources that have actually caused incidents rather than mandating coverage; make the check trivially cheap to add; and bring the incident record — “this change cost us eleven hours and the finance report was wrong for two days” is far more persuasive than a governance principle.

← Modelling, and why it still decides everythingData quality that is not a dashboard →