Apache Kafka and its competitors changed more between 2024 and 2026 than in the preceding five years, and most teaching material predates all of it. This chapter is the current mechanics and an unusually clear lesson about vendor benchmarks.

• Kafka 4.0 (March 2025) removed ZooKeeper entirely — removed, not deprecated. Metadata now lives in an internal consensus log, KRaft.
• The upgrade constraint that catches people: you cannot jump from a ZooKeeper cluster straight to 4.x. You must already be on 3.3 or later and already migrated to KRaft. Kafka 3.9 exists specifically as that bridge.
• Other breaks in the same release: the oldest message formats gone, old client protocol versions gone, and a Java version floor that differs between clients and brokers. Any material treating the coordination service as current is two major versions stale.

KIP-848 rewrote the consumer group rebalance protocol, and this is the most commonly misrepresented item in current material.

Tiered storage — hot segments on local disk, completed segments on object storage, invisible to producers and consumers — became production-ready in the last 3.x release. The real benefit is not storage cost: it is that brokers no longer copy cold data during reassignment, so scaling and rebalancing get dramatically faster.

The further step is diskless: write straight to object storage with no local replication at all. WarpStream and others ship it commercially and KIP-1150 was formally accepted upstream in March 2026 — but be precise here, because the marketing is not: it is a directional proposal, and the implementation is not in mainline. The only running implementations are vendor forks and managed products.

And the tradeoff nobody leads with: writing directly to object storage pushes produce latency into the hundreds of milliseconds. The economics are real — cross-zone network transfer and attached block storage are what you are eliminating, and one vendor's own price list shows storage dropping around 60% on its relaxed-latency tier. But it buys cost with latency, and any material presenting diskless as free is wrong.

The transactional machinery got a genuine correctness improvement: server-side defences are now automatically enabled, and with current clients the producer epoch is bumped on every transaction. That closes the hanging-transaction and zombie-write holes — which matter because a hanging transaction pins the last stable offset, which stalls every read-committed consumer and stops log compaction.

But be precise about what the guarantee covers. There are four mechanisms and they are frequently conflated:

• The idempotent producer deduplicates by producer, epoch and sequence — per partition, per producer session. It is not application-level deduplication.
• Transactions make a write across several partitions atomic together with the consumer offset commit.
• Read-committed consumers read only up to the last stable offset, skipping aborted batches.
• Fencing bumps the epoch when a transactional identifier is re-registered, which is how a zombie instance is stopped.
• What none of it gives you is exactly-once into a sink outside the system. ‘Exactly-once’ here means effectively-once within the platform's own boundary, and stating it that way is the answer that shows you have operated one.

And the commercial context worth knowing: IBM completed its acquisition of Confluent in March 2026, at an enterprise value around eleven billion dollars. Several of the interesting smaller companies were acquired in the same window. That does not change the protocol, and it changes your negotiating position.