Warren
Monitor

Alert rules

CommunityTeamPro

Warren samples every queue every 30 seconds. An alert rule says what to watch, what counts as a problem, and how long it has to last before anyone is told.

Four rules on the Alerts page: consumers gone, dead letters piling up, disk running low, mass failure
Four rules on the Alerts page: consumers gone, dead letters piling up, disk running low, mass failure

Start from a template

Alerts → Rules → New rule offers templates for what usually goes wrong. Pick one and adjust; the queue pattern is read off the naming of the cluster you are looking at (.dlq, .dead, _error …).

Templates for a new rule: dead letters piling up, a growing dead-letter queue, consumers gone, mass failure, disk, memory, publishers blocked, new failure
Templates for a new rule: dead letters piling up, a growing dead-letter queue, consumers gone, mass failure, disk, memory, publishers blocked, new failure
ConditionFires whenTypical use
Messages above thresholda queue holds more than N messagesdead letters piling up
Growth within a windowa queue gained N messages within the windowa dead-letter queue filling right now
Inflow above ratemore than N messages per minute arrivea mass failure, even while a replay rule keeps draining the queue
No consumersready messages wait and nobody consumesa crashed service on a work queue
Node: disk running lowfree disk above disk_free_limit is below N MBact before RabbitMQ blocks every publisher
Node: memory highmemory in use is above N % of the high watermarkthe same for memory
Node: resource alarma disk or memory alarm is onpublishers are blocked right now
New failurean exception or death reason turns up in a dead-letter queue that was not there beforethe one new failure among the familiar ones

Each rule matches queue names (or, for node conditions, node names) by regular expression, on one cluster or on all of them.

New failures

A dead-letter queue that always holds a few hundred familiar timeouts hides the one exception nobody has seen before. New failure looks at what the messages died of, not how many there are. While such a rule is enabled, Warren reads the dead-letter queues of its clusters every WARREN_FAILURE_SCAN_INTERVAL (2 minutes) and groups the messages as the Failures page does, by exception or by death reason.

What a queue holds the first time Warren reads it is known, not new. A failure that turns up after that fires the rule: "new failure in payments.dlq: InsufficientFundsException: account <n> (3 messages, first seen 14:02)". A second new failure while the alert fires is announced at once, not with the next reminder. The alert resolves when the new failures' messages are gone from the queue, or once they are older than the hours set on the rule (24 by default). A failure not seen for 30 days counts as new again when it returns.

Once, again, and all clear

For N seconds makes a condition hold for a while before the rule fires, so a short spike stays quiet. A firing alert notifies its channel once, reminds again after WARREN_ALERTING_RENOTIFY_AFTER (4 hours) while it lasts, and sends the all-clear when the queue or node recovers. The Events tab keeps the history: when it fired, the value, when it resolved.

Rules belong to admins; operators and viewers see them and their events.