Data Platform & Analytics
Data Engineer
The characteristic experience of the job is being paged for a change you did not make. Most of the craft is about making that stop.
Data engineering sits between systems that change without telling you and consumers who assume the numbers are right. Everything difficult about the role comes from that position.
| Time | What you are actually doing |
|---|---|
| 08:00 | A pipeline failed overnight. Ten minutes to establish whether it failed or, worse, half-succeeded — because a partial load looks like success and produces wrong numbers all day. |
| 09:30 | Root cause: a producing team renamed a field. Entirely reasonable from their side; nobody told them anyone depended on it. |
| 11:00 | Backfill. Reprocessing three days without double-counting, which is why the raw layer is append-only and never transformed in place. |
| 13:30 | Build. An incremental model to replace a full refresh that has quietly become the largest line on the warehouse bill. |
| 15:00 | An analyst asks why two dashboards disagree. They are computing 'active customer' differently, and the real fix is one definition in the core layer rather than reconciling these two. |
| 16:30 | Add a contract check to the source that broke this morning — in the producer's pipeline, not yours. |
The decisive round
“Your pipeline broke because upstream changed a schema. Prevent it.”
Interviewer: “A service team renamed a column and your pipeline broke. How do you stop this recurring?”
A weak answer. “I'd add schema validation to the pipeline so we detect it early, and set up alerting so we know as soon as it happens.” This detects faster and changes nothing structural. You are still the one paged, still after the change has shipped, and still explaining to finance why the report was wrong.
A strong answer. “Detecting it faster is worth doing and it does not solve the problem, because the change has already shipped and I am still the one paged for a decision someone else made.
The structural fix is a data contract whose check runs in the producer's pipeline. Their build goes red before merge, not mine at 3am. That single detail — whose CI fails — is the whole difference, because it moves the cost to the person who can prevent it.
The contract needs schema, but the part people leave out is semantics: what the field means, its unit, its currency. More breakages come from a silent unit change than from a type change, and no schema validator catches that.
Getting it adopted is political rather than technical, so I would not mandate coverage. I would start with the two or three sources that have actually caused incidents, make the check trivially cheap to add, and bring the incident record — 'this cost us eleven hours and finance was wrong for two days' persuades where a governance principle does not.
In the meantime, yes to detection: freshness and null-rate checks on the fields I depend on, because the contract will take a quarter and I need to survive this one. And I would keep the raw layer append-only and untransformed, so whatever happens I can reprocess.”
The fork is whose build fails. Candidates who add validation to their own pipeline have improved detection; candidates who move the check upstream have changed the incentive.
What to build
- A pipeline with a contract check that fails the producer's build, even if you simulate the producer. Being able to describe that mechanism is worth more than any tool name.
- An incremental model replacing a full refresh, with the before-and-after cost. Half of these postings name cost.
- A grain bug, deliberately created and found — a join that fans out and double-counts. Then the uniqueness test that catches it.
- A backfill that is safe to run twice. Idempotency is the property that makes recovery possible, and it gets asked about.
The recurring pattern in data incidents is not a failed job — it is a job that half-succeeded, reported green, and produced wrong numbers that were acted on for days.
Use it to say: “It is why my first check is volume against an expected band, not just success. A partial load looks exactly like a good one.”
Right-to-erasure requests routinely expose copies nobody had listed — a materialised mart, a backup, an analyst's extract.
Use it to say: “I use the erasure question as a lineage test: if we cannot answer it in an hour, our inventory is fiction.”
Week 1 — build
Days 1–3: A pipeline end to end on real messy data: raw append-only, staging, core, mart.
Days 4–5: Add the four checks — freshness, volume, uniqueness, null rate — and break each deliberately.
Week 2 — cost and correctness
Days 1–2: Convert a full refresh to incremental; measure the cost difference.
Days 3–5: Create a grain bug, find it with a test, and write up how it would have reached a dashboard.
Week 3 — rehearse
Days 1–2: The schema-change answer, out loud, ending on whose build fails.
Days 3–4: The two-dashboards-disagree answer, which is really about one definition in core.
Day 5: Five stories: an incident, a cost reduction, a contract you got adopted, a number you found wrong, a backfill.
Red flags
No raw layer. If transformations happen on ingest with nothing retained, you cannot reprocess and every bug is permanent.
Nobody owns definitions. If 'active customer' lives in six dashboards, you will spend your career reconciling numbers rather than building.
On-call for other teams' changes with no contract mandate. Ask whether you have any authority to require anything of producers. If not, the role is absorbing breakage.
Compensation
| Market | Band | Notes |
|---|---|---|
| United States | $140k – $230k base | Corpus p50 is $192k; platform-side data engineering runs above analytics-side |
| India | ₹15L – ₹50L total | Very large market; product companies pay well above services |
The book for this field
Data Platform & Analytics
What the job actually is now, modelling and grain, data contracts and who gets paged, quality that is not a dashboard, streaming and when you need it, governance and the 82%, cost, and serving the AI workload.
Cross-cutting
Skills every archetype tests
These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.
Practice questions across all themes → · Back to Data Platform & Analytics →