Warren
Act on dead letters

Replay dead letters

CommunityTeamPro

A replay puts messages from a dead-letter queue back into circulation. By default each one goes to the exchange and routing key it was originally published to, read from its x-death header, so it reaches the same consumers as before.

A dead-letter queue grouped by exception: 143 OrderNotFound, 41 PaymentDeclined, 16 timeouts
A dead-letter queue grouped by exception: 143 OrderNotFound, 41 PaymentDeclined, 16 timeouts

Choose the messages

On the queue page, group the dead letters by Exception, Reason, Died in, Routing key or Deaths and click a group to narrow the table. Then:

  • Selected in the table: tick the rows (x on the keyboard, Shift+A for all) and press Replay or r.
  • First N: the oldest N from the head of the queue.
  • Matching: every message that fits the search or the reason group, in the whole queue, not only the loaded rows.

"First N" and "Matching" are checked with a dry run first, so the replay takes exactly the messages you saw.

The replay dialog with two selected messages: original route, optional rate limit, and what happens to the headers
The replay dialog with two selected messages: original route, optional rate limit, and what happens to the headers

Choose the target

  • Original route from x-death, or from the x-original-* headers of Spring's RepublishMessageRecoverer, or the endpoint exchange of a MassTransit _error queue.
  • Queue: straight into a named queue through the default exchange.
  • Exchange with a routing key of your choice.

Select one message to edit its payload or headers before it goes out; the audit log keeps the original next to the edited version.

Go easy on the consumers

Limit to N messages per second spaces the publishes evenly, so a consumer that just recovered is not flooded again. A throttled replay runs in the background; its page shows the progress and can stop it. It may take at most 25 minutes, because RabbitMQ closes channels whose deliveries stay unacknowledged longer than its consumer_timeout.

What a replay promises

  1. Warren reads each message without acknowledging it.
  2. It publishes the message with mandatory and publisher confirms. Only after the broker confirms does it acknowledge the original.
  3. Everything it did not take goes back with nack and requeue; a crash releases it the same way.

No message is lost. If Warren dies between the broker's confirm and its own acknowledgement, the messages of that moment are delivered twice, so consumers should be idempotent, as RabbitMQ itself recommends.

Death headers (x-death, x-first-death-*, x-last-death-*), MassTransit's MT-Fault-* and a quorum queue's x-delivery-count are removed, so the message starts with a fresh retry budget. Warren adds x-warren-replay-id, x-warren-source-queue and x-warren-replayed-at, so you can tell a replayed message from a new one.

Every replay is in the Audit log with who, when, which route, and the outcome per message.