The fix is merged, the pipeline is green, and 143 messages wait in orders.dlq. Replaying all of them at once is a bet: that the new version runs on every consumer replica, that the fix covers these payloads and not only the one in the ticket, and that the consumer does not fail on something else further down. If the bet is lost, 143 messages die a second time, the alerts fire again, and the dead-letter queue now mixes old failures with new ones. Sending one message first and waiting a moment costs a couple of minutes and settles the question for almost nothing.
The idea as a video (canary replay and the incident report, captions, no sound needed):
A canary only means something if its message is processed during the wait. Check two things on the work queue first:
rabbitmqctl list_queues name messages consumers or the queue's page in the management UI. With no consumer the canary sits in the queue and every wait "passes".Then pick the canary from the failure you mean to fix. If the dead-letter queue holds two different exceptions, one canary passing says nothing about the other; send one per exception.
The replay is the same as for a whole batch, stopped after the first message: read it from the dead-letter queue, publish it to its original route with a header that names this replay, wait for the publisher confirm, then acknowledge the original.
import uuid, pika
DEATH = ("x-death", "x-first-death-", "x-last-death-", "x-delivery-count", "x-acquired-count")
replay_id = str(uuid.uuid4())
ch.confirm_delivery()
method, props, body = ch.basic_get("orders.dlq", auto_ack=False)
death = props.headers["x-death"][0] # broker dead-lettering; Spring or MassTransit error queues keep the route elsewhere
headers = {k: v for k, v in (props.headers or {}).items() if not k.startswith(DEATH)}
headers["x-replay-id"] = replay_id
props.headers = headers
ch.basic_publish(death["exchange"], death["routing-keys"][0], body, props, mandatory=True)
ch.basic_ack(method.delivery_tag) # confirm_delivery made the publish wait for the broker
print("canary", props.message_id, "replay", replay_id)Your own headers survive dead-lettering on classic and quorum queues, so if the canary dies again it comes back carrying x-replay-id, together with a fresh x-death. That pair is what tells its return apart from a new failure that arrived in the meantime. The details are in Did the replay work?.
Long enough for the consumer's whole retry chain to run out, plus the time to reach the canary. The chain is usually the larger part:
| Retry set-up | When a failing message reaches the DLQ |
|---|---|
No retries, basic.reject with requeue=false | within a second of delivery |
Spring RetryTemplate / @Retryable, 3 attempts, backoff 1 s × 2 | after about 3 s (1 + 2), plus processing |
| Broker-side retry: reject into a wait queue whose TTL dead-letters back, 5 rounds of 30 s | after about 2.5 minutes |
MassTransit UseDelayedRedelivery with 5, 15, 30 minutes | after about 50 minutes |
Read the numbers off your own configuration. Two minutes covers in-process retries with short backoffs; a delayed-redelivery set-up needs a canary that waits an hour, or a canary sent before lunch and the rest afterwards.
During the wait nothing else is replayed, so the dead-letter queue changes only through new failures and through the canary coming back. Check its depth; when it grows, look at what was added:
def came_back(ch, queue, replay_id, depth_before):
depth = ch.queue_declare(queue, passive=True).method.message_count
if depth <= depth_before:
return False
# returns arrive at the tail: read the whole queue and put everything back
found, tags = False, []
for _ in range(depth):
method, props, _ = ch.basic_get(queue, auto_ack=False)
if method is None:
break
tags.append(method.delivery_tag)
h = props.headers or {}
found = found or (h.get("x-replay-id") == replay_id and "x-death" in h)
for tag in tags:
ch.basic_nack(tag, requeue=True)
return foundTake depth_before right after the canary left the queue, and call this every few seconds until the wait is over. Reading and putting back marks messages as redelivered. On a quorum dead-letter queue with a delivery limit, check what a requeue costs on your RabbitMQ version before you peek at all.
If your consumer sends failures to a different queue than the one the message came from (a parking queue after the last retry, a Spring .dlq next to a broker DLQ), watch that queue too.
x-death: stop. Not back: replay the rest, throttled.