Lakshya

Data Platform & Analytics · Chapter 1 of 8

What the job actually is now

Compliance outranks SQL. Half the postings name cost. Real-time outnumbers batch seven to one.

3 min read1 diagramAll 8 chapters

Data engineering has changed faster than its own discourse. Parsing 76 data-platform job descriptions produces a ranking that does not look like the conference circuit.

76 DATA-PLATFORM JOB DESCRIPTIONS, FULL TEXT privacy & compliance 82%SQL 51%cost 50%streaming / Kafka 47%real-time 40%dbt 22%batch 6% Compliance outranks SQL. Half the postings name cost. Real-time outnumbers batch seven to one. None of that is what the data-engineering discourse talks about, and dbt — which dominates the conversation — appears in fewer than a quarter of postings.
Privacy and compliance appear in 82% of postings — more than SQL. Half name cost. Real-time outnumbers batch seven to one. And dbt, which dominates the conversation, appears in fewer than a quarter.

What that means for how you prepare

  • Governance is the job, not a constraint on it. Classification, residency, retention, lineage and access are named more often than any tool. If your answers are all about pipelines, you are answering a 2018 job description.
  • Cost is yours. Half of postings name it, and the warehouse bill is usually the second-largest line in a data organisation. Being unable to discuss it is a visible gap.
  • Streaming is the default assumption, at least in what postings say they want. Be able to reason about exactly-once, late arrivals and watermarks, even if most of your experience is batch — and be honest that most systems are still batch underneath.
  • Do not over-index on dbt. It is genuinely useful and it is not the differentiator its discourse share implies. Modelling judgement is; the tool is not.
← All chaptersModelling, and why it still decides everything →