Warren
Look at dead letters

Failures across queues

CommunityTeamPro

A queue page answers "why did the messages in this queue die?". Failures (g f) answers it for the whole cluster entry: every dead-letter queue is read, and the same exception in three queues is one row with three counts. That is how an outage of one downstream service shows up when it is used by several consumers.

Failures across the dead-letter queues of a cluster: timeouts, missing orders, declined payments with a team note, a tax service failure tagged new, grouped by exception
Failures across the dead-letter queues of a cluster: timeouts, missing orders, declined payments with a team note, a tax service failure tagged new, grouped by exception

How messages are grouped

  • By exception, from Spring's x-exception-*, x-last-exception-*, MassTransit's MT-Fault-* or a plain x-exception header (the same headers replay rules and the message view read): the exception's short type and its message with ids, numbers and hashes replaced by <n>, <uuid> and <hex>. These are the same groups as "why they died" by exception on a queue page, so a note on a group shows up here too. A note written under another breakdown (reason, routing key, died in, deaths) counts as well, as long as its group holds messages of the failure; the incident report lists it.
  • Without an exception, by death reason and the queue that dead-lettered it: "rejected in orders.process", "expired in payments.wait.5m".

Warren reads the first 200 messages of each dead-letter queue, the 30 fullest first; parking queues and quorum queues whose reads count as deliveries (RabbitMQ 3.13) are left out and named under the table. Each queue of a row is a chip with its count, which opens the queue; a chip also says how many of its messages carry no original route and shows a bookmark when the team left a note on the group there.

A failure that was not in its queue when Warren read the queue before is tagged new for a day, and each row says when Warren first saw it. An alert rule can tell you about new failures as they appear, see Alert rules.

Replay a failure everywhere

Replay on a row replays that failure's messages in every queue you tick, to each message's original route, optionally at most N per second per queue. A queue whose group carries a team note starts unticked, since a note usually says why not to. Messages without an original route stay where they are.

Each queue's replay is an ordinary replay with confirms and its own entry in the audit log; with four-eyes approval each waits for a second operator. Only the messages read here are taken, so a queue that holds more than 200 of them keeps the rest for the next round.

The incident report

Report on a row builds the story of that failure for the post-mortem: where and since when Warren has seen it, the team's notes, every replay, park and discard that touched its messages with who did it and what became of it (how many died again, whether a canary passed), and what is left in the queues now, as a timeline and as tables. Download Markdown (d) saves it as a .md file, Copy Markdown puts it on the clipboard; GET /api/clusters/{c}/failures/report?key= and …/report.md?key= return the same for scripts.

Nothing is stored for it: the report is built when you ask, from the audit log, the notes and the sightings Warren keeps for the "new" tag. An action is listed when its messages were recorded with their failure, which Warren does since 0.16; older entries of the audit log are not matched to a failure.