In Node the failure modes of a RabbitMQ consumer are quieter than elsewhere. An async handler that throws does not crash anything: the promise rejects, nobody awaits it, and the message stays unacknowledged until the channel closes. NestJS, with its defaults, acknowledges the message before your handler runs, so a failure simply loses it. And a nack with requeue puts the message straight back at the head of the queue. Here is how to make failed messages land in a dead-letter queue with amqplib and with NestJS, and how to send them back once the bug is fixed.
const amqp = require('amqplib');
const conn = await amqp.connect(process.env.AMQP_URL);
const ch = await conn.createChannel();
// the dead-letter route
await ch.assertExchange('orders.dlx', 'direct', { durable: true });
await ch.assertQueue('orders.dlq', { durable: true });
await ch.bindQueue('orders.dlq', 'orders.dlx', 'orders.dlq');
await ch.assertQueue('orders.process', {
durable: true,
deadLetterExchange: 'orders.dlx',
deadLetterRoutingKey: 'orders.dlq',
});
await ch.prefetch(20);
await ch.consume('orders.process', async (msg) => {
if (msg === null) return; // the broker cancelled the consumer, e.g. the queue was deleted
try {
await handle(JSON.parse(msg.content.toString()));
ch.ack(msg);
} catch (err) {
console.error('order failed, dead-lettering', msg.properties.messageId, err);
ch.nack(msg, false, false); // allUpTo=false, requeue=false
}
}, { noAck: false });
nack(msg, false, false) hands the message to the queue's dead-letter exchange, and RabbitMQ adds an x-death header with the queue, the reason (rejected), the time, and the original exchange and routing key. deadLetterExchange and deadLetterRoutingKey are amqplib's names for the x-dead-letter-* queue arguments.
What goes wrong without this:
consumer_timeout (30 minutes by default) RabbitMQ closes the channel and requeues them, and it starts again.nack(msg) with the defaults. The third argument defaults to requeue=true. The message goes back to the head of the queue and is delivered again at once: a hot loop that burns CPU and logs. It does not end on a classic queue, and not on a quorum queue either: measured on RabbitMQ 4.3.6, a basic.nack with requeue does not count against the delivery limit, 1,000 rounds took under three seconds and the message was still there. Only a consumer that dies holding the message counts (and ch.reject(msg) with requeue, which also moves it to the back).ack, or an ack on a different channel, is a protocol error: the broker closes the channel with PRECONDITION_FAILED - unknown delivery tag, and every unacknowledged message on it is requeued. Keep the ack on one code path.JSON.parse outside the try. A malformed body is the classic poison message. It belongs in the dead-letter queue, not in a crash.Queue arguments are fixed when a queue is created. If orders.process already exists without them, assertQueue fails with PRECONDITION_FAILED and closes the channel. For existing queues use a policy:
rabbitmqctl set_policy orders-dlx '^orders\.process$' \
'{"dead-letter-exchange":"orders.dlx","dead-letter-routing-key":"orders.dlq"}' \
--apply-to queues
The RabbitMQ transport of @nestjs/microservices acknowledges every message on delivery unless you set noAck: false. With the default, a handler that throws logs an error and the message is already gone. Turn manual acknowledgement on and declare the dead-letter route on the queue:
// main.ts
app.connectMicroservice<MicroserviceOptions>({
transport: Transport.RMQ,
options: {
urls: [process.env.AMQP_URL],
queue: 'orders.process',
noAck: false,
prefetchCount: 20,
queueOptions: {
durable: true,
arguments: {
'x-dead-letter-exchange': 'orders.dlx',
'x-dead-letter-routing-key': 'orders.dlq',
},
},
},
});
// orders.controller.ts
@EventPattern('order.created')
async onOrderCreated(@Payload() order: OrderCreated, @Ctx() context: RmqContext) {
const channel = context.getChannelRef();
const message = context.getMessage();
try {
await this.orders.process(order);
channel.ack(message);
} catch (err) {
this.logger.error(`order ${order.id} failed, dead-lettering`, err);
channel.nack(message, false, false);
}
}
The exchange orders.dlx and the queue orders.dlq are not declared by Nest; create them once with amqplib as above, with rabbitmqadmin, or in a definitions file. Messages emitted through a Nest ClientProxy carry Nest's envelope as their body ({"pattern": "order.created", "data": {...}}); that is what the handler matches on, so keep the body exactly as it is when you replay.
Retrying inside the handler with setTimeout keeps the message unacknowledged in your process and loses the attempt count on a restart. Let the broker hold it instead: orders.process dead-letters into a wait queue orders.retry.wait with x-message-ttl: 30000 and no consumers, whose own dead-letter exchange routes back to orders.process. Every round increments the count of the same x-death entry, so the handler can give up after a few:
function rounds(msg, queue) {
const deaths = msg.properties.headers?.['x-death'] ?? [];
const entry = deaths.find((d) => d.queue === queue && d.reason === 'rejected');
return entry ? Number(entry.count) : 0;
}
// in the catch block
if (isTemporary(err) && rounds(msg, 'orders.process') < 5) {
ch.nack(msg, false, false); // to the wait queue, back in 30 s
} else {
await park(msg); // publish to orders.dlq on a confirm channel
ch.ack(msg);
}
amqplib decodes the AMQP long values in x-death as numbers; Number() keeps it safe if your setup returns them as strings or BigInts. Paste a message into the x-death decoder to see how such a loop reads.
Read from the dead-letter queue, publish to the original route on a confirm channel, acknowledge the dead-lettered copy only after the broker confirmed:
const DEATH = ['x-death', 'x-first-death-', 'x-last-death-', 'x-delivery-count', 'x-acquired-count'];
const ch = await conn.createConfirmChannel();
for (;;) {
const msg = await ch.get('orders.dlq', { noAck: false });
if (!msg) break;
const [death] = msg.properties.headers['x-death'];
const headers = Object.fromEntries(
Object.entries(msg.properties.headers).filter(([k]) => !DEATH.some((p) => k.startsWith(p))),
);
ch.publish(death.exchange, death['routing-keys'][0], msg.content, { ...msg.properties, headers, mandatory: true });
await ch.waitForConfirms();
ch.ack(msg);
}
Replay one message first and watch whether it comes back before sending the rest; Replay one message first describes how long to wait and what a pass proves.
noAck: false, a try/catch around the whole handler including parsing, nack(msg, false, false) on failure.nack(msg): its default requeues.noAck: false, ack and nack through RmqContext, dead-letter arguments in queueOptions.x-death, not in setTimeout.x-death, death headers stripped, one message first.