Data Platform & Analytics · Chapter 1 of 16
What the job actually is now
Compliance outranks SQL. Half the postings name cost. Real-time outnumbers batch seven to one.
Data engineering has changed faster than its own discourse. Parsing 76 data-platform job descriptions produces a ranking that does not look like the conference circuit.
What that means for how you prepare
- Governance is the job, not a constraint on it. Classification, residency, retention, lineage and access are named more often than any tool. If your answers are all about pipelines, you are answering a 2018 job description.
- Cost is yours. Half of postings name it, and the warehouse bill is usually the second-largest line in a data organisation. Being unable to discuss it is a visible gap.
- Streaming is the default assumption, at least in what postings say they want. Be able to reason about exactly-once, late arrivals and watermarks, even if most of your experience is batch — and be honest that most systems are still batch underneath.
- Do not over-index on dbt. It is genuinely useful and it is not the differentiator its discourse share implies. Modelling judgement is; the tool is not.
The four jobs that share one title
“Data engineer” covers at least four different jobs. They interview differently, pay differently and lead to different places, and the fastest way to waste a job search is to prepare for one and apply to another.
| The job | What you actually do daily | What the interview tests |
|---|---|---|
| Pipeline builder | Ingest, transform, schedule, fix what broke overnight | SQL, Python, a warehouse, orchestration. The most common posting by a wide margin |
| Analytics engineer | Model the warehouse so analysts can self-serve | Modelling and semantics — grain, slowly-changing dimensions, metric definitions. Much less infrastructure than people expect |
| Data platform engineer | Build the systems other data teams use | Distributed systems, cost, multi-tenancy, reliability. This is a software infrastructure job and pays like one |
| Streaming / real-time | Event pipelines with latency and correctness requirements | Watermarks, state, exactly-once boundaries. The smallest and most specialised pool |
The compensation observation worth acting on: the highest-paying signals in this field are distributed-systems and infrastructure skills, not warehouse-tool proficiency. Tool names appear in postings at high frequency and are the least differentiating thing on a CV, because everyone applying has them. The scarce thing is being able to reason about a system under failure and under cost pressure.
The three questions that identify which job a posting is
- “Who consumes what this team builds?” Analysts and dashboards → analytics engineering. Other data engineers → platform. Applications and customers → a software engineering job wearing a data title.
- “What is the on-call situation?” No on-call usually means analytics engineering. A real rotation means you own systems, and the interview will reflect that.
- “What is the biggest source of toil right now?” The honest answer — and most teams give one — tells you what the first year is. ‘Pipelines break and we find out from Slack’ is a very different year from ‘we are migrating off a legacy warehouse’.
Check yourself