Infrastructure & Reliability · Middleware — Messaging · Chapter 26 of 33
Kafka and IBM MQ: a log is not a queue
The comparison bank messaging interviews ask most: why a log that remembers differs from a queue that forgets, what Kafka’s ‘exactly once’ does and does not cover, and what changed when Kafka learned to queue in February 2026.
Apache Kafka and IBM MQ are both called messaging, both move data between systems, and both appear on the same architecture diagrams in every large bank. They are built on different ideas, and almost every design mistake made with either comes from treating it as the other.
This chapter is the comparison, not a Kafka tutorial. Kafka’s internals — its coordination layer, storage tiers and the 2025 release that removed ZooKeeper — are covered in streaming internals; the corpus behind this site shows Kafka in 4.5% of job descriptions, the only technology in this whole section with a measurable presence in it.
The one idea: a queue forgets, a log remembers
Kafka divides each topic into partitions — separate logs that can live on different servers, which is how it scales. Order is guaranteed within a partition, not across them. A producer chooses the partition by a key: key every event by account number and all of one account’s events land in one partition, in order, which is the same ‘order per key’ principle as in the basics.
The consequences, side by side
| Property | IBM MQ — a queue | Kafka — a log |
|---|---|---|
| After a message is consumed | Deleted | Kept until the retention period ends |
| Replay | Not possible, unless you copied it somewhere | Reset a group’s offset and read again |
| Many independent readers | Publish/subscribe gives each subscriber a copy | Natural: every consumer group has its own offset, and readers do not affect each other |
| Ordering | Per queue, with one consumer; per group of messages | Per partition |
| Scaling consumers | Add consumers to the queue | Classic groups: at most one consumer per partition. Share groups (from 4.2): more consumers than partitions |
| Acknowledgement | Per message | Classic groups: commit an offset, covering everything before it. Share groups: per record |
| Transactions with a database | Yes — XA two-phase commit | No XA. Kafka transactions cover writes to Kafka and the consumer’s offsets, nothing outside |
| What it is optimised for | Per-message guarantees, low latency, mainframe integration | Very high throughput in batches, retention, many readers |
| Holds history | No | Yes — the log can be the system’s memory of events |
What Kafka’s ‘exactly once’ covers
Kafka offers two mechanisms that together are marketed as exactly-once semantics:
- The idempotent producer. Each producer gets an identifier and numbers its messages, so when a network glitch makes it resend, the broker recognises and discards the duplicate.
- Transactions. A producer can write to several partitions and record the consumer offsets it read from, atomically: all of it becomes visible, or none of it.
Together these make a read from Kafka, process, write to Kafka pipeline exactly-once. They say nothing about anything outside Kafka. If the processing step updates a database, sends an email or calls a payment switch, a crash between that action and the offset commit will repeat it. For those cases the answer is the same as for any broker: an idempotent consumer, or the outbox pattern. Saying “Kafka has exactly-once, so duplicates cannot happen” in a bank interview is a reliable way to end the conversation early.
The settings that make Kafka durable
# producer
acks=all # wait until every in-sync replica has the write
enable.idempotence=true # broker discards resent duplicates
# topic
replication.factor=3 # three copies, on three brokers, ideally three zones
min.insync.replicas=2 # refuse writes unless at least two copies are current
unclean.leader.election.enable=false # never promote a replica that is behindFebruary 2026: Kafka learns to queue
For most of its life, Kafka could not do one thing a queue does trivially: let many workers share the messages of one partition, acknowledging each one individually. A consumer group could never have more active members than partitions, so parallelism was fixed by how the topic was laid out.
Queues for Kafka changed that. Proposed as KIP-932 (a Kafka Improvement Proposal), it introduced share groups: consumers that cooperatively read the same partitions, with per-record acknowledgement, a delivery count for each record, and automatic setting-aside of records that keep failing — the Kafka equivalent of a backout threshold. It shipped as early access in Kafka 4.0, as a preview in 4.1, and was declared production-ready in Apache Kafka 4.2.0 on 17 February 2026, which also added a way to extend the processing time on a record.
What share groups do not change
“Kafka is not a queue” is now an out-of-date sentence, and an interviewer may test whether you know that. But share groups do not add XA transactions, do not integrate with CICS (the Customer Information Control System, IBM's mainframe transaction server), and do not change the fact that the data stays in the log for its retention period. The decision rules below still hold; one argument for running two systems has become weaker.
Where each belongs in a bank
| Use IBM MQ when… | Use Kafka when… |
|---|---|
| The message is a command that must be executed exactly once by one system — a payment instruction to the core | The message is an event many systems care about — “payment posted” for fraud, alerts, analytics |
| The other end is CICS, the mainframe, or an interface that mandates it — SWIFT’s MQ adaptor | The other end is a data platform, a stream processor or a microservice estate |
| You need the unit of work to span the message and a database (XA) | You need replay — a new service that must see the last seven days, or a reprocessing after a bug |
| Volumes are high but per-message latency and certainty matter most | Volumes are very high and throughput matters most |
| The data should disappear once processed | The event history is itself valuable — audit, rebuilding a read model, feeding models |
Most large banks run both, joined by connectors that copy messages from IBM MQ queues into Kafka topics and back. The shape that has emerged is MQ for the transactional spine, Kafka for the nervous system around it — the flows chapter traces both.
One owner for both
On 17 March 2026 IBM completed its acquisition of Confluent, the main commercial Kafka company, for $31 a share in cash, about $11 billion. IBM therefore owns the dominant bank queue manager and the leading commercial Kafka distribution. Apache Kafka itself remains an Apache Software Foundation project that anyone can run for free; what changed is who sells most of the support, the managed service and the tooling around it. For a bank’s procurement team, that shifts negotiating leverage; for an architect, it removes one reason a vendor would argue for one system over the other.
Bridging the two, and moving a flow across
A connector that copies messages from an IBM MQ queue into a Kafka topic is a consumer on one side and a producer on the other, so it inherits the weaker of the two sides’ guarantees unless it is specifically configured for more. Treat the bridge as at-least-once and make the Kafka consumers idempotent — the same rule as everywhere else in this sub-section.
When a bank moves a flow from IBM MQ to Kafka, four things change that are easy to miss in a design review:
- Acknowledgement. Per-message acknowledgement becomes offset commits, unless the flow uses share groups.
- Poison handling. There is no backout queue by default; the consumer must route failing records to a dead-letter topic itself, or rely on share groups’ delivery counting.
- Ordering. Order per queue becomes order per partition, so the partition key must match the business ordering key — the account, not a random value.
- Retention — the one with legal consequences. A queue forgets; a log is a data store. Moving payment messages to Kafka means personal data now sits in a system with a retention period, subject to the bank’s data-retention rules, deletion requests and encryption-at-rest requirements. A transient pipe has become a database, and its governance must follow.
Check yourself