Infrastructure & Reliability · Middleware — Messaging · Chapter 24 of 33
What a message queue is, and why banks cannot live without one
From a phone call to a posted letter: what asynchronous means, what a broker actually guarantees, and the vocabulary — queues, topics, acknowledgements, dead letters — that every other messaging chapter assumes.
Two programs can talk in one of two ways. The first is a phone call: the caller dials, waits, and cannot do anything else until the other side answers — and if the other side is not there, the call fails. The second is a posted letter: the sender drops it in a post box and carries on with the day, trusting the postal service to deliver it whenever the recipient is home.
The web tier works like a phone call. Messaging works like the post — and the software that plays the postal service is called a message broker (IBM’s term, used across banks, is a queue manager). Everything else in this sub-section is detail on top of that one idea, so it is worth seeing why it matters so much to a bank.
That is the whole case for messaging in a bank, in three properties:
- Decoupling in time. The sender and receiver do not need to be running at the same moment. A mainframe that is unavailable during its overnight batch does not stop the mobile app from accepting instructions.
- Buffering. A burst of 50,000 payments at 9am does not need a receiver that can process 50,000 at once; the queue absorbs the burst and the receiver drains it at its own pace. The arithmetic of the thread pool no longer applies across the boundary.
- Guaranteed delivery. A message handed to the broker will arrive, even if a network or a server fails in between — provided it is configured to be persistent, which is the first thing to check in any design review.
The vocabulary
| Term | What it means |
|---|---|
| Message | A unit of data with a header (who sent it, when, its identifier, where replies go) and a body or payload — a payment instruction, an account event |
| Producer / consumer | The program that sends; the program that receives. The same program is often both |
| Queue | A named, ordered holding place; each message is taken by one consumer |
| Topic | A named channel for publishing; every subscriber receives a copy |
| Broker / queue manager | The server that owns queues and topics, stores messages and moves them |
| Persistent message | Written to disk before the broker confirms it, so it survives a restart. A non-persistent message is faster and is lost if the broker stops |
| Acknowledgement | The consumer’s confirmation that it has finished with a message, after which the broker may delete it |
| Correlation identifier | A value copied from a request into its reply, so the requester can match them |
| Expiry | A time after which an unconsumed message is discarded — right for a price quote, dangerous for a payment |
| Dead-letter queue | Where messages go when they cannot be delivered or processed |
Two shapes: queues and topics
What ‘guaranteed’ really guarantees
Interviewers probe this more than anything else in messaging, because the words are used loosely. There are three possible delivery guarantees:
| Guarantee | What it means | How it is achieved | Where it is acceptable |
|---|---|---|---|
| At most once | Never duplicated, possibly lost | Send and forget; no acknowledgement | Telemetry, price ticks — the next one supersedes it |
| At least once | Never lost, possibly duplicated | The broker redelivers until it is acknowledged; a consumer that crashes after acting but before acknowledging gets the message again | Anywhere the consumer can recognise a duplicate |
| Exactly once | Never lost, never duplicated | Either a transaction that commits the consumer’s work and the acknowledgement together, or at-least-once delivery plus a consumer that ignores duplicates | Payments — and it is always partly the application’s job |
The rule that survives every broker and every vendor claim
Design every consumer to be idempotent — meaning that processing the same message twice has the same effect as processing it once. In practice: every payment carries a unique business identifier, and the consumer records identifiers it has already applied and skips repeats.
A broker can guarantee what happens inside the broker. It cannot know whether your database update committed before your process died. Idempotency is what makes the end-to-end guarantee true regardless of where the failure falls.
Transactions: the unit of work
A consumer typically does two things with a message: it updates a database, and it tells the broker it is finished. If it does one and dies before the other, you get either a lost payment or a duplicate. There are three ways to make the pair safe:
- A distributed transaction. A transaction manager coordinates the database and the broker with a two-phase commit (2PC): both first promise they can commit, then both commit. The standard for this is XA, from the X/Open group. IBM MQ and full Java application servers support it; it is correct, and it is slow and operationally delicate.
- The outbox pattern. The application writes its business change and the outgoing message into the same database in one ordinary transaction; a separate process relays the outbox to the broker. No distributed transaction is needed, and the relay can safely retry.
- At-least-once plus idempotency. Accept that a message may be processed twice, and make the second time harmless. The simplest, and in modern systems the most common.
Order, and why banks care about it
A queue is first-in, first-out — for a single consumer. Add a second consumer to share the work and two messages can finish in either order. For most messages that does not matter. For a bank it matters exactly where it hurts: a debit and a credit to the same account, applied in the wrong order, can bounce a payment that should have succeeded.
The standard answer is to guarantee order per key rather than globally: all messages for one account go to the same consumer, in order, while different accounts are processed in parallel. IBM MQ does this with message groups; Kafka does it with partitions, covered in Kafka versus MQ.
Poison messages and the dead-letter queue
Sooner or later a message arrives that the consumer cannot process — malformed, referring to an account that does not exist, or triggering a bug. The consumer fails, the transaction rolls back, the message returns to the queue, and the consumer takes it again, forever. A poison message stops everything queued behind it.
The defence is a backout threshold: after a set number of failed attempts, the message is moved aside to a backout or dead-letter queue so the rest can flow. That solves the technical problem and creates an organisational one: a dead-letter queue is a pile of payments nobody processed. Someone must own it, be alerted when it is not empty, and reconcile every message in it. An unattended dead-letter queue is the most common way a bank ‘loses’ a payment that the middleware delivered perfectly.
Request and reply over a queue
Messaging is asynchronous, but plenty of bank traffic needs an answer — a balance, an authorisation. The pattern is request/reply: the requester puts a message on a request queue with a reply-to queue name in the header, then waits on that reply queue for a message whose correlation identifier matches. The replier processes the request and puts the answer where it was told. It is a phone call built out of two letters, and it inherits a letter’s problem: a reply can arrive after the requester has stopped waiting, a problem the banking flows chapter treats at length.
How deep is too deep: queue depth as the vital sign
A queue’s depth — how many messages are waiting — is the first number to watch. Rising depth means consumers are slower than producers; a queue at its maximum depth refuses new messages, and the producer receives an error (IBM MQ reports it as reason code 2053, queue full). The better alert is often the age of the oldest message: a depth of 10,000 that drains every minute is healthy; a depth of 3 that has not moved for an hour is an incident.
Protocols and interfaces
| Name | What it is |
|---|---|
| MQI | IBM MQ’s own programming interface, the Message Queue Interface — calls such as MQPUT and MQGET |
| JMS | Jakarta Messaging, formerly the Java Message Service: a Java API, not a network protocol. Code written to it can run against IBM MQ, Apache Artemis or others |
| AMQP | The Advanced Message Queuing Protocol — an open wire protocol; version 1.0 is an international standard (ISO/IEC 19464) |
| MQTT | Originally MQ Telemetry Transport, created at IBM in 1999: a very light publish/subscribe protocol for devices and unreliable networks, now an OASIS standard |
| Kafka protocol | Kafka’s own binary protocol, implemented by Kafka-compatible services from several vendors |
Check yourself