Lakshya

Data Platform & Analytics

Data Platform Engineer

Data · 21 postings

Builds the substrate the data engineers use. The paved road for data, with the same adoption problem every platform has.

Where a data engineer builds pipelines, a data platform engineer builds the thing pipelines are built on — ingestion framework, orchestration, the warehouse itself, the catalogue, the access model and the cost controls. Your customers are internal and they can build their own if yours is worse.

TimeWhat you are actually doing
09:00Warehouse cost review. One team's pipeline is 40% of the bill and they do not know, because nothing attributes spend.
10:30Ingestion framework work — making a new source a config file rather than a bespoke pipeline.
13:00Access model. Deriving grants from classification instead of from tickets, so nobody has to ask.
14:30A team has built their own scheduler because ours was awkward. The interesting question is which part was awkward.
16:00Upgrading the orchestrator across every pipeline, with a codemod rather than forty tickets.

The decisive round

“Design a data platform for 200 engineers and 50 analysts.”

Interviewer: “Greenfield. What do you build first?”

A weak answer. “I'd set up a lakehouse with a medallion architecture, an orchestrator, a transformation layer, a catalogue and a BI tool, with CI/CD for the pipelines.” A reference architecture recited, in an order determined by a vendor diagram rather than by what unblocks anyone.

A strong answer. “I would not build the architecture first. I would find the two or three things people wait for, because that is what determines whether anyone uses it.

Usually it is: getting a new source in, getting access to data they can see is there, and knowing whether a number is trustworthy. So the first thing I build is ingestion as configuration — adding a source is a file, not a bespoke pipeline — because that removes the largest queue and buys the credibility to do everything else.

Then access derived from classification rather than from tickets, so the answer to ‘can I see this’ is automatic and the security team gets a model instead of a backlog.

Then the layered model — raw append-only, staging, core, marts — because without it we get six definitions of customer and the platform becomes a place numbers are argued about.

Then cost attribution from day one. Not optimisation — attribution. Tag warehouses and queries by team, because without it nobody owns anything and every cost conversation is about the total.

What I would deliberately defer: the catalogue, which is worth little until there is something worth cataloguing, and the semantic layer, which people build early and then find nobody adopts. And I would measure adoption — percentage of sources on the framework, time from request to data available — because a platform nobody uses is indistinguishable from one that does not exist.”

Building for the queue rather than the diagram, deferring the catalogue, and treating cost attribution as foundational rather than an optimisation phase.

  • Build ingestion-as-config for two different source types and show the second one took an hour.
  • Implement classification-derived access on something real, then test it with two users of different privilege.
  • Produce a cost-per-team report from billing data. The unattributed percentage is always the finding.
  • Have an adoption number and its trend, because this is a platform role and adoption is your revenue equivalent.

Compensation

MarketBandNotes
United States$150k – $250k baseAbove pipeline-side data engineering
India₹18L – ₹55L totalProduct companies and GCCs building central data platforms

The book for this field

Data Platform & Analytics

What the job actually is now, modelling and grain, data contracts and who gets paged, quality that is not a dashboard, streaming and when you need it, governance and the 82%, cost, and serving the AI workload.

CHAPTER 1What the job actually is nowCompliance outranks SQL. Half the postings name cost. Real-time outnumbers batch seven to one.READ THE CHAPTER →CHAPTER 7Cost, because half the postings name itThe warehouse bill is usually the second-largest line in a data organisation, and it is yours.READ THE CHAPTER →CHAPTER 3Data contracts, and who gets pagedThe structural fix for the defining problem of the field: your pipeline breaks because of a change you did not make.READ THE CHAPTER →

All 8 chapters in Data Platform & Analytics →

Cross-cutting

Skills every archetype tests

These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.

Design under constraintThe decisive round for SRE, AI Platform, Infrastructure — and the reason strong engineers fail it.READ THE CHAPTER →Measuring what resists measurementTested in every field on this site under a different name — evals, SLOs, DORA, ablations. Named in 59% of AI and 42% of platform postings, and almost nobody studies it deliberately.READ THE CHAPTER →The controlOne habit separates strong candidates from plausible ones more reliably than any technical depth.READ THE CHAPTER →

Practice questions across all themes →  ·  Back to Data Platform & Analytics →