Warren
Look at dead letters

Queues and why messages died

CommunityTeamPro

The queue list

Queues lists every queue of the cluster entry selected in the sidebar. Dead-letter queues come first, with their message count in red.

The queue list: dead-letter queues on top, messages, consumers, rates, when each queue is empty, and where its dead letters go
The queue list: dead-letter queues on top, messages, consumers, rates, when each queue is empty, and where its dead letters go

Warren recognises a dead-letter queue by its binding to an exchange that another queue uses as x-dead-letter-exchange, or by its name (dlq, dlx, dead-letter, .dead, .failed, MassTransit's _error and _skipped; WARREN_DLQ_NAME_PATTERN changes the list). Tags mark the special ones:

TagMeaning
DLQcollects dead letters
WAITa retry queue with a TTL that sends messages back to work; empty ones stay hidden until you search for them
PARKEDa parking queue Warren moves poison messages to
LIMITa quorum queue whose every peek counts as a delivery (RabbitMQ 3.13), see Quorum queues

Empty in says when a queue is empty at the current pace, from the depth of the last ten minutes. Columns hides what you do not need.

Why they died

Open a queue to read its messages without taking them. The panel on top groups the loaded messages:

  • Reason: rejected, expired, maxlen, delivery_limit, or republished when a consumer moved the message itself.
  • Died in: the queue that dead-lettered it.
  • Routing key it was published with.
  • Exception the consumer recorded (Spring's x-exception-*, MassTransit's MT-Fault-*), with numbers and ids replaced by <n> so equal failures fall into one group.
  • Deaths: how often it went round already.
orders.dlq grouped by exception: 143 OrderNotFoundException, 41 PaymentDeclinedException, 16 SocketTimeoutException
orders.dlq grouped by exception: 143 OrderNotFoundException, 41 PaymentDeclinedException, 16 SocketTimeoutException

Click a group to narrow the table to it. Peek sets how many messages are read (up to 200 at a time); the rest of the queue is reachable through a matching selection.

Notes on a group

Some dead letters are known: a bug with a fix on the way, a bank that declines cards. Operators can put a note on a group, and everyone sees it under that bar, in each message of the group, and at the top of the replay dialog when a replay would take such messages.

A note on the PaymentDeclinedException group: declined by the bank, do not replay
A note on the PaymentDeclinedException group: declined by the bank, do not replay

Click the speech bubble at the end of a bar to write one; a note belongs to one group in one queue and stays when the messages are gone. The Failures page and its incident report show it for every failure whose messages are in that group, whichever breakdown you wrote it in.

The search field (/) finds text in payloads, message and correlation ids, routing keys, fingerprints and header values, where consumers leave their exception messages. A search can be applied to the whole queue as a matching selection in the replay, discard and park dialogs.

Flow and the settings in force

Flow (v) draws the queue with the exchanges that publish into it, the applications consuming it, and where its dead letters go, whether the dead-letter exchange is set by argument or by policy.

Under the graph, Settings in force answers why a queue dead-letters where it does, or not at all:

  • the policy that applies, with its pattern, priority and apply-to, and the operator policy if there is one;
  • policies that match the queue's name as well but are outranked. RabbitMQ applies one policy per queue, the one with the highest priority, so a second policy with a dead-letter exchange does nothing, and Warren says so;
  • each dead-letter and limit setting (dead-letter-exchange, dead-letter-routing-key, dead-letter-strategy, message-ttl, expires, max-length, max-length-bytes, overflow, delivery-limit) with the value in force and where it comes from, plus the values that are set but lose.

The rules are RabbitMQ's: a queue argument beats the policy for the dead-letter keys and overflow; for the limits the lowest of argument, policy and operator policy wins, and an unlimited delivery limit (-1) loses to any set one. A quorum queue without a delivery limit has RabbitMQ 4's default of 20.