Data Platform & Analytics · Chapter 5 of 8
Streaming, and when you actually need it
Named in 47% of postings against 6% for batch. Here is what that does and does not mean.
Streaming language dominates the postings, and the honest position is that a large share of systems described as real-time are micro-batch underneath and that is usually correct. Knowing when the complexity is justified is a stronger signal than enthusiasm for it.
The three hard problems, which no tool removes
- Late and out-of-order events. A mobile client goes offline and sends its events four hours later, into a window you already closed and reported on. Watermarks and allowed lateness are how you decide, explicitly, what to do — and whatever you choose, some correct data is discarded or some published number changes retroactively.
- Exactly-once is a property of the whole path, not of the broker. A pipeline with exactly-once processing and a non-idempotent sink is at-least-once, and idempotent writes are usually a simpler answer than transactional semantics.
- State grows. A streaming job holding windows and joins accumulates state that must be checkpointed and restored, and a job that cannot restart quickly from state is a job that cannot be operated.
The question to ask before building it
“What decision gets made differently because this is seconds fresh rather than fifteen minutes fresh?”
For fraud, trading, operational alerting and personalisation there is a real answer. For a dashboard a human reads twice a day there is not, and you have taken on state management, replay complexity and a much harder debugging story for nothing. The strongest answer in an interview names a case where you argued against streaming.