Failures across queues
A queue page answers "why did the messages in this queue die?". Failures (g f) answers it for the whole cluster entry: every dead-letter queue is read, and the same exception in three queues is one row with three counts. That is how an outage of one downstream service shows up when it is used by several consumers.
How messages are grouped
- By exception, from Spring's
x-exception-*,x-last-exception-*or MassTransit'sMT-Fault-*headers: the exception's short type and its message with ids, numbers and hashes replaced by<n>,<uuid>and<hex>. These are the same groups as "why they died" by exception on a queue page, so a note on a group shows up here too. - Without an exception, by death reason and the queue that dead-lettered it: "rejected in orders.process", "expired in payments.wait.5m".
Warren reads the first 200 messages of each dead-letter queue, the 30 fullest first; parking queues and quorum queues whose reads count as deliveries (RabbitMQ 3.13) are left out and named under the table. Expand a row to see each queue's share and open the queue.
Replay a failure everywhere
Replay on a row replays that failure's messages in every queue you tick, to each message's original route, optionally at most N per second per queue. A queue whose group carries a team note starts unticked, since a note usually says why not to. Messages without an original route stay where they are.
Each queue's replay is an ordinary replay with confirms and its own entry in the audit log; with four-eyes approval each waits for a second operator. Only the messages read here are taken, so a queue that holds more than 200 of them keeps the rest for the next round.