Data Platform & Analytics · Chapter 5 of 16
Streaming, and when you actually need it
Named in 47% of postings against 6% for batch. Here is what that does and does not mean.
Streaming language dominates the postings, and the honest position is that a large share of systems described as real-time are micro-batch underneath and that is usually correct. Knowing when the complexity is justified is a stronger signal than enthusiasm for it.
The three hard problems, which no tool removes
- Late and out-of-order events. A mobile client goes offline and sends its events four hours later, into a window you already closed and reported on. Watermarks and allowed lateness are how you decide, explicitly, what to do — and whatever you choose, some correct data is discarded or some published number changes retroactively.
- Exactly-once is a property of the whole path, not of the broker. A pipeline with exactly-once processing and a non-idempotent sink is at-least-once, and idempotent writes are usually a simpler answer than transactional semantics.
- State grows. A streaming job holding windows and joins accumulates state that must be checkpointed and restored, and a job that cannot restart quickly from state is a job that cannot be operated.
The question to ask before building it
“What decision gets made differently because this is seconds fresh rather than fifteen minutes fresh?”
For fraud, trading, operational alerting and personalisation there is a real answer. For a dashboard a human reads twice a day there is not, and you have taken on state management, replay complexity and a much harder debugging story for nothing. The strongest answer in an interview names a case where you argued against streaming.
When you actually need streaming, and when you have been told you do
Streaming is the most over-adopted architecture in data engineering. The honest test is not ‘is the data continuous?’ — almost all data is. It is ‘does a decision get made on this data faster than a batch would deliver it?’
| Situation | Verdict |
|---|---|
| A dashboard refreshed hourly that people look at daily | Batch. Streaming here buys nothing and costs an always-on system with its own failure modes |
| Fraud scoring on a transaction in flight | Streaming. The decision happens inside the event |
| Operational alerting on system behaviour | Streaming — or, quite often, your monitoring system, which already does this |
| Feeding a machine-learning feature store | Depends entirely on inference latency. If the model runs nightly, the features can be built nightly |
| "The business wants real-time" | Ask what decision changes. Very often the answer is that a report arrives at 8am instead of 9am, which is a scheduling change |
The cost that is never in the proposal: a streaming pipeline is a permanently running distributed system with state, watermarks, backpressure and its own on-call burden. A batch job that fails is rerun in the morning. A streaming job that fails is an incident, and its state may not be recoverable if nobody thought about checkpointing.
The middle option almost nobody considers: micro-batch on a short interval.
Running a batch job every five minutes gives you five-minute freshness with batch semantics — idempotent reruns, simple failure handling, no watermarks, no state to recover, no exactly-once conversation.
For a very large share of ‘we need real-time’ requirements, that is sufficient and it is a fraction of the operational cost.
The threshold where genuine streaming becomes necessary is roughly when the requirement is sub-minute and a decision depends on it. Above that, frequent batch usually wins on total cost of ownership even where it looks less impressive on a diagram.
So the sequence to propose is: daily batch, then hourly, then a short interval, then streaming — and only move a step when something concrete fails. Most teams jump to the last option because it is the one people write about.
Check yourself