Warren
Look at dead letters

Set-up checks and the dead-letter map

CommunityTeamPro

Most lost dead letters are not lost in an incident but in the set-up: a queue that never got a dead-letter exchange, an exchange that was renamed, a DLQ nobody watches. Warren reads the queues, exchanges and bindings of the cluster entry and says what it finds, on top of the queue list. It changes nothing in the broker.

Set-up checks on the queue list: a quorum DLQ that can drop dead letters, with the policy that fixes it
Set-up checks on the queue list: a quorum DLQ that can drop dead letters, with the policy that fixes it

What it checks

FindingWhy it matters
A queue with consumers but no dead-letter exchangea rejected, expired or over-limit message is gone without a trace
A dead-letter exchange that does not existevery dead letter of the queue is dropped
A dead-letter exchange no queue is bound tothe same: dead letters arrive and are dropped
A quorum dead-letter queue with a delivery limita message arrives with its delivery count and one crash of whatever reads the DLQ drops it, see Quorum queues
A dead-letter queue no alert rule watches (Team, Pro)dead letters pile up unnoticed

Each finding comes with the fix, usually a policy you can paste into rabbitmqctl. The dead-letter exchange is read from the queue's arguments and from its effective policy. Temporary queues (exclusive, auto-delete, amq.*) and parking queues are left out.

The dead-letter map

Dead-letter map above the checks draws where the dead letters of the cluster entry go, left to right: queues that dead-letter, the exchanges they use, the queues those route to, retry loops through wait queues, and the parking queues Warren moves messages to. A dead-letter exchange that drops messages is red. Click a queue to open it.

The dead-letter map: a retry loop through orders.wait.5m, orders.process dead-lettering to orders.dlq and its parking queue, and two more chains
The dead-letter map: a retry loop through orders.wait.5m, orders.process dead-lettering to orders.dlq and its parking queue, and two more chains

Queues that take no part in dead-lettering stay off the map, so it shows the dead-letter wiring of a cluster with hundreds of queues on one screen. GET /api/clusters/{c}/checks and GET /api/clusters/{c}/dead-letter-map return the same for scripts.