Data Platform & Analytics · Chapter 12 of 16
Stream processing: watermarks, state and the engines
The concepts that transfer, the two major-version breaks that stranded people, and why one engine's micro-batch reputation is now out of date.
Stream processing is where the genuinely hard ideas in data engineering live — event time, watermarks, and distributed state — and where interview questions actually differentiate. This chapter is the mechanics first, the engines second.
Event time versus processing time
Event time is when the thing happened. Processing time is when your operator saw it. Everything difficult in streaming follows from the gap between them, and only event time gives you results that reproduce when you replay.
Watermarks, and the most common production bug
A watermark is a heuristic assertion, not a guarantee. It says ‘I believe no event older than T will arrive from now on’. Treating it as a guarantee is the single most common streaming bug in production.
They propagate as a minimum. In a distributed engine, watermarks are generated per source partition and combined by taking the minimum across all input channels — so one idle or lagging partition holds back the entire pipeline, and every window downstream stops emitting. Configure idleness detection and watermark alignment, or you will meet this at three in the morning.
Late data is silently dropped unless you configure a side output for it. That default has quietly lost a great deal of data in a great many companies.
And one popular engine has no watermarks at all. The in-process stream library uses stream time — which advances monotonically with observed record timestamps — plus a per-window grace period. It is a genuinely different mental model and should not be taught as equivalent.
Checkpointing, and why alignment matters under load
The distributed engine takes consistent snapshots by injecting barriers at the sources that flow with the data. An operator with several inputs waits for the barrier on all of them — alignment — before snapshotting its state.
Under backpressure, alignment is what makes checkpoints time out, which is the classic production failure: the job is running, throughput is fine, and checkpoints keep failing until the job cannot recover. That is why unaligned checkpoints exist — they snapshot in-flight buffers rather than waiting, trading checkpoint size for reliability. Knowing when to turn them on is a genuinely useful piece of operational knowledge.
And know the distinction between checkpoints — engine-owned, tuned for fast recovery, possibly in a native format — and savepoints, which are user-owned and portable. It matters because the newest state backend does not support portable savepoints, which is an operational trap for anyone adopting it.
The two major-version breaks
Both leading engines shipped breaking major versions and both stranded people:
- Apache Flink 2.0 (March 2025), its first major release in roughly eight years, removed the batch dataset API, all Scala APIs, the older source and sink interfaces, and the per-job deployment mode. State is not guaranteed compatible across the major version, which is the actual migration blocker — and it is why the last 1.x release is maintained as long-term support and why a large share of production is still on it.
- Its headline architectural change is disaggregated state: a new backend that puts state files on a distributed filesystem rather than local disk, paired with asynchronous execution so operators can do parallel non-blocking state access. State size becomes bounded by object storage rather than by node disk, and checkpoints become very light. Note the evidence quality here is better than usual — there is a peer-reviewed conference paper describing the design, which is a different class of evidence from a throughput claim.
- Apache Spark's micro-batch reputation is now out of date. Spark 4.1 introduced a Real-Time Mode with sub-second latency and single-digit-millisecond processing for stateless work. Any material asserting ‘that engine is micro-batch only’ is wrong as of that release, and it is one of the most commonly repeated stale claims in the field.
- It also replaced its arbitrary-stateful operators with a new abstraction offering composite state types, automatic time-based eviction and timers — a much better model than the map-groups-with-state functions people learned.
The engines, and what each is actually for
| Engine | Model | Choose it when |
|---|---|---|
| In-process library | A JAR in your application, scaled by adding instances; partitions are the parallelism | You want stream processing without a cluster, your logic is Kafka-to-Kafka, and stream-time semantics are enough |
| Distributed engine | Long-running job graph with checkpointed state | You need genuine event-time semantics, large state, complex windows, or exactly-once state across many operators |
| Batch engine's streaming mode | Incremental execution on a batch engine, now with a real-time mode | Your team already runs batch on it and the streaming work is an extension rather than a new discipline |
| Incremental view maintenance | Maintain a SQL view as data changes, propagating retractions | The under-appreciated category. If what you want is 'this query, always current', this is a much smaller commitment than a stream processor |
That last row deserves a note. Two products in the category take different bets: one is permissively licensed with state pushed to object storage and a Postgres wire protocol; the other is source-available and cloud-only, built on a differential-dataflow foundation that handles genuinely hard query shapes including recursion, with state in memory.
Neither is a replacement for a general stream processor — they are incremental view maintenance engines with a SQL surface. But a large share of what teams build with a full stream processor is, in fact, a materialised view that needs to stay current, and recognising that can remove an entire system from your architecture.
The commercial picture, which consolidated sharply
Worth knowing because it affects support and roadmaps rather than the code: every significant managed offering in this space now sits inside a large acquirer. The main commercial vendor behind the distributed engine belongs to a cloud giant; the two independent managed-service startups were both acquired, one by a streaming platform vendor now itself owned by a very large incumbent, and one by a database company. A promising open-source SQL-first engine was acquired and its release velocity dropped noticeably afterwards.
The honest framing: vendor-neutral engine, consolidated vendors. The project governance is genuinely multi-party; the commercial ecosystem is not.
Check yourself