Every RabbitMQ operator has the management UI open, and when a dead-letter queue fills up, it is the first place anyone looks. Each queue page has a small corner for messages: Get messages, Move messages, Publish message and Purge. It is enough for a lot of days. It also has a few defaults and side effects that are easy to miss in the middle of an incident. Here is what each button does, measured on RabbitMQ 4.3.6 with classic and quorum queues.
The same task in the management UI and in Warren, side by side (2:02, captions, no sound needed):
Get messages fetches messages with basic.get and then settles them according to the Ack Mode you pick. The form starts at one message, and payloads are cut at 50,000 bytes.
| Ack Mode | What happens to the message |
|---|---|
| Nack message requeue true (default) | put back, marked redelivered |
| Automatic ack | removed from the queue |
| Reject requeue true | put back, marked redelivered |
| Reject requeue false | removed, or dead-lettered again if the queue has a dead-letter exchange |
The two "put back" modes look the same. On a classic queue they are: both put the message back where it was. On a quorum queue they are not:
quorum DLQ, three messages m1 m2 m3, then one Get messages of m1:
Nack message requeue true -> m1 m2 m3, x-delivery-count of m1 unchanged
Reject requeue true -> m2 m3 m1, x-delivery-count of m1 + 1Measured on 4.3.6, also with plain AMQP: basic.nack with requeue=true does not count against a quorum queue's delivery limit, basic.reject with requeue=true does, and it moves the message to the back. Every Reject requeue true spends one delivery of the message's budget.
The two "removed" modes are the dangerous ones on a dead-letter queue. A DLQ usually has no dead-letter exchange of its own, so Reject requeue false does not move the message anywhere: it is gone, exactly like Automatic ack.
Use the default. Nack message requeue true is the only mode that neither removes a message nor costs it a delivery.
Even the default has one trap. Since RabbitMQ 4.0 every quorum queue has a delivery limit of 20 unless you set one, and a message that a quorum work queue dead-letters for delivery_limit arrives in a quorum DLQ with its count already at 21. On 4.3.6, a message whose count is above the queue's limit is dropped the moment it is put back, and Get messages with requeue puts it back. One look and it is gone.
Before anyone opens Get messages on a quorum DLQ, take the limit off:
rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queuesThe quorum article has the measurements and two other fixes.
Get messages shows the headers of each message as nested tables. The x-death table is the incident report: the queue the message died in, the reason (rejected, expired, maxlen, delivery_limit), how often, and the exchange and routing keys it was originally published with. The x-death article goes through it field by field.
What the UI does not do is look across messages. With five messages you read them one by one. With 300 messages and three different exceptions, you are scrolling a page of raw payloads and nested tables, looking for the dozen that are different. There is no search inside a queue and no grouping.
Move messages appears on the queue page once the rabbitmq_shovel and rabbitmq_shovel_management plugins are enabled. It asks for one thing, a destination queue, and creates a dynamic shovel named Move from <queue> with these settings:
src-delete-after: queue-length # move what is there now, then delete the shovel
ack-mode: on-confirm # a message leaves the source once the destination has it
dest-queue: <what you typed>
dest-add-forward-headers: falseThat makes it a safe move: nothing is lost on the way, and the shovel removes itself when it is done. What it does not do matters more for dead letters:
x-death; Move messages does not read them.x-death, x-first-death-*, x-last-death-* and, from a quorum queue, the old x-delivery-count as a plain header. A consumer that reads the x-death count to decide when to give up sees the old count on arrival. If yours does, strip the headers when you replay instead.When the destination is the one queue the messages should go to, and you trust all of them, it is the right button.
To send a single message back to its route, people copy the payload from Get messages into Publish message on the exchange. Two things get lost on the way. The form sets properties and headers you type in, and its own help says it: "Only long string headers can be set here." Numbers, booleans and nested headers do not survive. And the original stays in the DLQ until you remove it with a second, destructive Get messages. Copy, publish, check, then remove, in that order.
Purge deletes every message in the queue. The UI asks once, in its own words: "Are you sure? Messages cannot be recovered after purging." There is no copy, no list of what was in the queue and no record of who did it. If a queue might hold anything you need later, export it first (Get messages with Nack message requeue true and a high count, then save the page, or a script).
The management UI keeps no record of messages fetched, moved, published or purged. If the postmortem asks who replayed what at 3:12, the answer is in someone's shell history or nowhere.
delivery-limit: -1 before the first peek.