Data Platform & Analytics · Chapter 6 of 16
Governance, privacy and the 82%
Named in more postings than SQL. Treat it as the job rather than as paperwork.
Privacy and compliance language appears in 82% of the data-platform postings behind this site. That is not because these teams love policy — it is because a data platform is where an organisation's regulatory exposure physically lives.
The five things you will actually be asked to implement
| Requirement | What it means in the platform | Where it usually fails |
|---|---|---|
| Classification | Every column tagged by sensitivity, ideally at ingestion | Applied to tables, not columns, so everything inherits the strictest tag |
| Access control | Row and column-level, derived from classification | Granted per-person ad hoc; nobody can answer who can see salary data |
| Retention | Automatic deletion on a schedule per class | Nothing is ever deleted, because deletion is scary and nobody owns it |
| Erasure | Find and remove one person across every copy | Backups, exports, and the analyst's local extract |
| Residency | Data physically stays in a jurisdiction | A replica or a backup quietly crosses a border |
The question that finds the real problem
“If a customer asks us to delete their data tomorrow, what exactly do we do?” The honest answer usually reveals four copies nobody had listed — the warehouse, a materialised mart, a backup, and someone's extract. Erasure is the requirement that exposes whether the lineage and the inventory are real, which is why it is worth asking first.
Make the platform the easy path
Governance imposed as a review gate produces shadow pipelines. Governance implemented as defaults — classification applied at ingestion, access derived from it, retention scheduled automatically — produces compliance without anybody choosing it.
This is the same principle as the paved road in platform engineering, and it is the difference between a governance programme that works and one that generates documents.
Governance that people comply with
Data governance fails in the same way everywhere: a policy is written, a tool is bought, a committee meets, and the actual behaviour of the organisation does not change. The programmes that work share one property — the governed path is the easiest path.
- Classification only works if it is applied at creation, by the person who knows what the data is. A retrospective classification exercise across an existing estate is a project that never finishes, and everyone involved knows it by month three.
- Access requests should be faster through the governed route than around it. If a self-service request takes two days and asking an engineer takes ten minutes, you have designed a shadow process and then blamed people for using it.
- Retention has to be automated to be real. A policy saying data is deleted after seven years, with no mechanism, means you are holding everything forever and have documented that you should not be — which is a worse legal position than having no policy.
- Lineage earns its place operationally, not for the auditor. Its value is answering ‘this number is wrong, what feeds it’ and ‘we are changing this column, what breaks’ under time pressure.
- And ownership must be a person, not a team. A dataset owned by a team is owned by nobody the moment a question needs answering in an hour.
The two questions a governance programme should be able to answer
“Where is our personal data, and who can reach it?” Almost no organisation can answer this in less than several days, and it is the question every regulator and every breach begins with.
“If a customer asks us to delete their data, what happens?” Trace it end to end. The answer is usually incomplete in the same three places: backups, analytical copies in the warehouse, and anything that reached a fine-tuned model — where deletion means retraining rather than a delete statement.
If you can answer both, your governance is real regardless of what the policy document says. If you cannot, no amount of committee structure changes that.
And the practical starting point is smaller than most programmes assume: a named owner and a classification for the twenty datasets that matter most, with the deletion path traced for the ones holding personal data. That is a few weeks of work, and it answers more real questions than a full-estate catalogue exercise that runs for a year and finishes at 60% coverage.
Check yourself