Data Platform & Analytics
Streaming / Real-time Engineer
The specialism with the hardest failure modes, and the one where knowing when not to use it is the differentiator.
Streaming language appears in 47% of these postings against 6% for batch. A large share of systems described as real-time are micro-batch underneath and that is usually correct — so the valuable engineer here is the one who can tell the difference.
| Time | What you are actually doing |
|---|---|
| 08:30 | Consumer lag climbing on one partition. Either a hot key or a slow downstream, and the distinction matters. |
| 10:00 | A late-arriving batch of events from a mobile client that was offline. The window closed yesterday and someone reported on it. |
| 13:00 | State size. The job's checkpoint has grown to the point where restart takes twenty minutes, which means it cannot be operated. |
| 15:00 | Investigating a duplicate. Exactly-once processing, non-idempotent sink, and therefore at-least-once in practice. |
| 16:30 | Arguing, correctly, that a requested real-time dashboard should be five-minute micro-batch. |
The decisive round
“Events arrive four hours late. What happens to your numbers?”
Interviewer: “A mobile client goes offline and sends its events four hours later. You have already closed and reported that window. Now what?”
A weak answer. “I'd configure a longer allowed lateness so the window stays open and the events are included.” Solves the immediate case and quietly commits you to holding state four hours longer for every window, and it does not address what happens to the number you already published.
A strong answer. “There is no free answer here, and I would say that first — every option discards correct data or changes a published number, and the job is to choose deliberately rather than to have the framework choose for me.
Concretely there are three. Drop them beyond a watermark, which keeps the numbers stable and means we are knowingly slightly wrong, and the size of the error is measurable if I count what I dropped. Hold the window open longer, which costs state proportional to the lateness allowance for every window, not just this one — a four-hour allowance on a one-minute window is a very different system. Or accept restatement: publish, then correct, which is honest and requires that every consumer downstream can tolerate a number changing, which most dashboards and no commission calculation can.
What I would actually do depends on the consumer. For operational alerting, drop and count. For finance, restate — they already understand restatement and would rather be correct late than stable and wrong. For a general dashboard, a bounded allowance plus a visible ‘as of’ timestamp.
And I would instrument the lateness distribution rather than guessing, because the allowance should come from data. Usually the tail is much longer than anyone assumes and the ninety-fifth percentile is a far better basis than a round number someone picked.”
Knowing there is no free option, tying the choice to the consumer, and measuring the lateness distribution instead of picking a round number.
- Build a windowed aggregation and deliberately inject late events. Watch what each lateness setting does.
- Break exactly-once by using a non-idempotent sink, then fix it with idempotent writes rather than transactions.
- Grow state until restart is slow, then measure recovery time. That number is what makes a job operable or not.
- Have a case where you argued against streaming and were right.
Compensation
| Market | Band | Notes |
|---|---|---|
| United States | $150k – $260k base | Scarcity-priced; genuine streaming operators are uncommon |
| India | ₹18L – ₹55L total | Concentrated in fintech, adtech and consumer-scale companies |
The book for this field
Data Platform & Analytics
What the job actually is now, modelling and grain, data contracts and who gets paged, quality that is not a dashboard, streaming and when you need it, governance and the 82%, cost, and serving the AI workload.
Cross-cutting
Skills every archetype tests
These are shared across every field on this site — the same question asked in different vocabulary — so preparation here compounds rather than being spent once. The book above is specific to your field.
Practice questions across all themes → · Back to Data Platform & Analytics →