Trading & Capital Markets
Market Data Engineer
Feed handlers, tick storage, and the corporate-actions problem that silently invalidates other people's research.
Feed handlers, tick storage, and the corporate-actions problem that silently invalidates other people's research.
A day that is actually representative
| Time | What you are actually doing |
|---|---|
| 07:30 | Pre-open: instrument master refresh, expiry rolls, new listings, and today's corporate actions. |
| 09:15 | Open. Watching for gaps, sequence-number breaks and a feed that has silently stopped rather than failed. |
| 11:30 | A vendor's tick file disagrees with the exchange's own end-of-day file. Somebody is wrong and you have to determine who. |
| 14:00 | Capacity: the options chain has grown again and the tick store is not keeping up. |
| 16:00 | Building the point-in-time view, so a researcher asking ‘what did we know at 10:31’ gets a truthful answer. |
The decisive round
“A researcher says your data is wrong.”
Interviewer: “Their backtest broke. Walk me through it.”
A weak answer. “I'd check the data pipeline for errors, compare against the exchange's official files, and look at whether there was an outage or a gap during that period.” Reasonable and incomplete. The most common cause of ‘the data is wrong’ is not missing data — it is data that is present, correct, and being used with the wrong time semantics.
A strong answer. “My first question would be which of three quite different things they mean, because they get reported identically.
Missing. A gap, a sequence break, an outage. This is the easiest to check and the least common cause of a broken backtest — and it's visible: compare bar counts against the exchange's own end-of-day file.
Wrong. A bad print, a vendor error, a mis-mapped instrument. Real, and usually detectable by cross-checking two sources on the disputed day.
Correct but used wrongly. This is nearly always the actual answer and it's the one worth being good at.
The specific traps, in the order they occur: adjusted prices. If they used a split- and dividend-adjusted series, the adjustment was applied retroactively to the whole history — so the price they 'saw' on a date in 2019 embeds information from a corporate action in 2023. That is lookahead, and it is invisible because the series looks perfectly clean. For India this bites constantly with bonus issues and splits.
Timestamp semantics. Is this the exchange timestamp, the vendor's receipt timestamp, or our capture timestamp? They can differ by milliseconds intraday and by a whole bar at the boundary. And is a bar labelled with its open or its close? An off-by-one-bar error is the single most common source of a spectacular fake backtest.
Instrument identity over time. A symbol is not an identity. Companies rename, merge, get delisted and re-list; a futures contract's symbol rolls monthly. If they joined on symbol rather than on a stable instrument ID with validity dates, they have silently stitched two different instruments into one series.
Survivorship in the universe. If the universe list came from today, everything that failed is missing.
What I'd build so this stops recurring: a point-in-time API where every query takes an as-of timestamp and returns exactly what was knowable then — including the reference data, not just the prices. Once researchers cannot accidentally get the future, most of this class disappears. And a daily reconciliation against the exchange's official files with an alert, so I find the problem before a researcher does.”
Separating missing from wrong from misused, and knowing that adjusted prices are a lookahead vector. The point-in-time API is the systemic fix and it is what a senior data engineer in this field is actually for.
What to build or bring
- Handle a real feed. Even a broker WebSocket. Sequence gaps, reconnection, and snapshot-plus-increments recovery.
- Know a columnar tick store — Parquet plus DuckDB is enough to demonstrate the ideas; kdb+/q is still the institutional standard and knowing it is a genuine differentiator.
- Be able to explain corporate actions and why adjusted series are dangerous for research.
- Understand the protocols: FIX for orders, ITCH/OUCH-style binary for data, and multicast versus TCP delivery.
Compensation
| Market | Band | Notes |
|---|---|---|
| United States | $150k – $350k | Hedge funds and prop firms above vendors |
| India | ₹15L – ₹60L | Strong demand at brokers, prop firms and GCC trading-tech teams |
The book for this field
Trading & Capital Markets
Eight chapters on how markets actually work: the order lifecycle end to end, the limit order book and how a price is made, the Indian and US market structures side by side, what each instrument is really for, the full cost of a trade, market data and the errors that ruin research, and the latency budget.
Also read for this role
Banking & Capital Markets Systems
What Indian banks and financial institutions actually run — core banking, payments, lending, treasury, custody, the registrar duopoly and regulatory reporting — and, just as usefully, which of it can be verified at all.
Cross-cutting
Skills every archetype tests
These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.
Practice the 10 Trading questions → · Back to Trading & Capital Markets →