Lakshya

Trading & Capital Markets

Market Data Engineer

Trading · the foundation everything sits on

Feed handlers, tick storage, and the corporate-actions problem that silently invalidates other people's research.

Feed handlers, tick storage, and the corporate-actions problem that silently invalidates other people's research.

A day that is actually representative

TimeWhat you are actually doing
07:30Pre-open: instrument master refresh, expiry rolls, new listings, and today's corporate actions.
09:15Open. Watching for gaps, sequence-number breaks and a feed that has silently stopped rather than failed.
11:30A vendor's tick file disagrees with the exchange's own end-of-day file. Somebody is wrong and you have to determine who.
14:00Capacity: the options chain has grown again and the tick store is not keeping up.
16:00Building the point-in-time view, so a researcher asking ‘what did we know at 10:31’ gets a truthful answer.

The decisive round

“A researcher says your data is wrong.”

Interviewer: “Their backtest broke. Walk me through it.”

A weak answer. “I'd check the data pipeline for errors, compare against the exchange's official files, and look at whether there was an outage or a gap during that period.” Reasonable and incomplete. The most common cause of ‘the data is wrong’ is not missing data — it is data that is present, correct, and being used with the wrong time semantics.

A strong answer. “My first question would be which of three quite different things they mean, because they get reported identically.

Missing. A gap, a sequence break, an outage. This is the easiest to check and the least common cause of a broken backtest — and it's visible: compare bar counts against the exchange's own end-of-day file.

Wrong. A bad print, a vendor error, a mis-mapped instrument. Real, and usually detectable by cross-checking two sources on the disputed day.

Correct but used wrongly. This is nearly always the actual answer and it's the one worth being good at.

The specific traps, in the order they occur: adjusted prices. If they used a split- and dividend-adjusted series, the adjustment was applied retroactively to the whole history — so the price they 'saw' on a date in 2019 embeds information from a corporate action in 2023. That is lookahead, and it is invisible because the series looks perfectly clean. For India this bites constantly with bonus issues and splits.

Timestamp semantics. Is this the exchange timestamp, the vendor's receipt timestamp, or our capture timestamp? They can differ by milliseconds intraday and by a whole bar at the boundary. And is a bar labelled with its open or its close? An off-by-one-bar error is the single most common source of a spectacular fake backtest.

Instrument identity over time. A symbol is not an identity. Companies rename, merge, get delisted and re-list; a futures contract's symbol rolls monthly. If they joined on symbol rather than on a stable instrument ID with validity dates, they have silently stitched two different instruments into one series.

Survivorship in the universe. If the universe list came from today, everything that failed is missing.

What I'd build so this stops recurring: a point-in-time API where every query takes an as-of timestamp and returns exactly what was knowable then — including the reference data, not just the prices. Once researchers cannot accidentally get the future, most of this class disappears. And a daily reconciliation against the exchange's official files with an alert, so I find the problem before a researcher does.”

Separating missing from wrong from misused, and knowing that adjusted prices are a lookahead vector. The point-in-time API is the systemic fix and it is what a senior data engineer in this field is actually for.

What to build or bring

  • Handle a real feed. Even a broker WebSocket. Sequence gaps, reconnection, and snapshot-plus-increments recovery.
  • Know a columnar tick store — Parquet plus DuckDB is enough to demonstrate the ideas; kdb+/q is still the institutional standard and knowing it is a genuine differentiator.
  • Be able to explain corporate actions and why adjusted series are dangerous for research.
  • Understand the protocols: FIX for orders, ITCH/OUCH-style binary for data, and multicast versus TCP delivery.

Compensation

MarketBandNotes
United States$150k – $350kHedge funds and prop firms above vendors
India₹15L – ₹60LStrong demand at brokers, prop firms and GCC trading-tech teams

The book for this field

Trading & Capital Markets

Eight chapters on how markets actually work: the order lifecycle end to end, the limit order book and how a price is made, the Indian and US market structures side by side, what each instrument is really for, the full cost of a trade, market data and the errors that ruin research, and the latency budget.

CHAPTER 7Market data: what it is, what it costs, and who sells itThe largest line item in most trading technology budgets, and the source of the errors that quietly invalidate research.READ THE CHAPTER →CHAPTER 2The order book, the spread, and how a price is actually madePrice is not a number the market has. It is the outcome of a queue, and understanding the queue explains costs, liquidity and most of what a trading firm does.READ THE CHAPTER →CHAPTER 1What actually happens when you press buyFrom a tap to a settled share, in both markets. The structural difference between India and the US is bigger than people expect, and it changes what the jobs are.READ THE CHAPTER →

All 8 chapters in Trading & Capital Markets →

Also read for this role

Banking & Capital Markets Systems

What Indian banks and financial institutions actually run — core banking, payments, lending, treasury, custody, the registrar duopoly and regulatory reporting — and, just as usefully, which of it can be verified at all.

CHAPTER 6Trading, and the exchange as gatekeeper of your algorithmWhy SEBI's approval regime surprises engineers from other markets, and what actually connects to NSE and BSE.READ THE CHAPTER →CHAPTER 7Custody, depositories, and the settlement fact everyone gets wrongCDSL and NSDL do not split the market the way people assume, and India has not moved to T+0.READ THE CHAPTER →CHAPTER 5Treasury, FX, and the single counterparty underneath all of itWhy CCIL has no analogue in most engineers' mental model of markets, and why the vendor question matters less than you would think.READ THE CHAPTER →

All 13 chapters in Banking & Capital Markets Systems →

Cross-cutting

Skills every archetype tests

These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.

Measuring what resists measurementTested in every field on this site under a different name — evals, SLOs, DORA, ablations. Named in 59% of AI and 42% of platform postings, and almost nobody studies it deliberately.READ THE CHAPTER →The controlOne habit separates strong candidates from plausible ones more reliably than any technical depth.READ THE CHAPTER →Failure and incident narrativeEvery archetype asks for this story. The outage is the setting; the decisions are the content.READ THE CHAPTER →

Practice the 10 Trading questions →  ·  Back to Trading & Capital Markets →