A dead-letter queue is supposed to be the place where failed messages are safe until someone looks at them. On RabbitMQ 4 with quorum queues and default settings, that is not quite true: a message that died because it was delivered too often can arrive in the DLQ with its delivery budget already used up, and the first consumer or tool that crashes while holding it makes RabbitMQ drop it for good. Here is what happens, measured on RabbitMQ 4.3.6, and three ways to make the DLQ safe again.
Quorum queues count how often a message was delivered and not settled. Past the queue's delivery limit (x-delivery-limit, or delivery-limit in a policy) RabbitMQ stops redelivering it: the message is dead-lettered with reason delivery_limit if the queue has a dead-letter exchange, and dropped otherwise. Since RabbitMQ 4.0 every quorum queue has a limit of 20 unless you set one, which is good: a poison message no longer loops forever.
What counts changed in 4.0 as well:
| What happens to a delivered message | 3.13 | 4.x |
|---|---|---|
basic.nack / basic.reject with requeue=true | counts | does not count |
| management UI "Get messages" with requeue | counts | does not count |
| the channel or connection closes while it is unacknowledged (consumer crash, pod killed, network drop) | counts | counts |
basic.reject with requeue=false (dead-lettered) | not measured | counts |
The count is visible on the message as the header x-delivery-count (RabbitMQ 4 adds x-acquired-count, every acquisition including requeues).
A work queue and a dead-letter queue, both quorum queues, no limits set, so both have RabbitMQ 4's default of 20:
ch.exchange_declare('lim.dlx', 'fanout', durable=True)
ch.queue_declare('lim.dlq', durable=True, arguments={'x-queue-type': 'quorum'})
ch.queue_bind('lim.dlq', 'lim.dlx')
ch.queue_declare('lim.work', durable=True,
arguments={'x-queue-type': 'quorum', 'x-dead-letter-exchange': 'lim.dlx'})
ch.basic_publish('', 'lim.work', b'order-4711')Then a consumer that crashes on this message every time: fetch it without acknowledging and close the connection, until the work queue gives up.
while message_count('lim.work') > 0:
conn = pika.BlockingConnection(params)
conn.channel().basic_get('lim.work', auto_ack=False)
conn.close() # the consumer "crashed" while holding the messageThe results:
abandoned deliveries on lim.work until it gave up: 21
in lim.dlq: reason delivery_limit, x-delivery-count 21
left in lim.dlq after ONE abandoned delivery there: 0The message reached the DLQ, as it should, with reason delivery_limit. But it brought its count of 21 along, and the DLQ's own limit is 20. The next abnormal return, a single one, made the DLQ give up on it too, and since the DLQ has no dead-letter exchange of its own, the message was dropped.
The delivery count is not just a header the queue writes for you to read: it is part of the message's annotations, and dead-lettering keeps them. The DLQ continues counting from where the work queue stopped. A header that a publisher sets is a different thing: we published a fresh message with x-delivery-count: 5 into a queue with a limit of 5, and RabbitMQ ignored it. The next delivery showed 1, and the message survived five abandoned deliveries. So the carried count comes from dead-lettering, not from the header itself.
It is not only delivery_limit messages. In the same test with explicit limits, a message that had been abandoned twice and then rejected arrived in the DLQ with a count of 3, and with a DLQ limit of 5 it was dropped after three more failures, not five. Every message rejected by a consumer arrives with at least 1.
More often than you would think:
basic.get and dies, or is stopped with Ctrl+C, before it acknowledges or nacks;On 3.13 it was worse (every requeue counted, so even looking at messages in the management UI cost deliveries), but 4.x's default limit makes the carry-over bite with no configuration at all.
All three kept the message through three abandoned deliveries in the DLQ in our test.
1. No delivery limit on the DLQ. A DLQ is not consumed in a loop, so it does not need one. By policy:
rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queuesor with x-delivery-limit: -1 when you declare the queue; a negative value means unlimited. Only one policy applies to a queue: if one matches your DLQs already, add the key there.
2. A much higher limit, if you want a backstop anyway: x-delivery-limit: 100.
3. A classic queue as DLQ. Classic queues have no delivery limit. You lose the replication of a quorum queue, which matters less for a queue that is mostly written and rarely read; decide by how much you care about dead letters surviving the loss of a node.
And in whatever reads the DLQ: settle every message you fetch, ack what you have handled and nack with requeue=true what you have not; on 4.x a requeue costs nothing.
x-delivery-count next to x-death. A value close to the DLQ's limit means the message is one or two crashes away from being lost.rabbitmqctl list_queues name type arguments policy shows which DLQs are quorum queues and whether a limit is set; no x-delivery-limit and no delivery-limit policy means 20.delivery-limit: -1 by policy, or classic DLQs.x-delivery-count before you leave dead letters lying around for days.