closing AMQP connection <0.3846.0> (192.168.65.1:60870 -> 172.17.0.7:5672 - billing-kill, vhost: 'orders', user: 'app', duration: '2s'):
client unexpectedly closed TCP connection
The TCP connection ended without the AMQP connection.close handshake. Inside the broker the reason is called connection_closed_abruptly (that is the name people search for); the log prints it as client unexpectedly closed TCP connection, at warning level. RabbitMQ 3.7.28, 3.8.34, 3.13.7 and 4.3.6 all wrote the same line. It says the client side went away, not why. Measured on 4.3.6 with pika 1.4.4:
| What happened | Broker log |
|---|---|
client process killed (kill -9) | client unexpectedly closed TCP connection |
| a proxy with a 10-second idle timeout between client and broker, heartbeat 60 s | client unexpectedly closed TCP connection, after 10 s |
| the same proxy, heartbeat 4 s | nothing: the connection was still open after 30 s |
| client blocked in a 40-second job, heartbeat 5 s | missed heartbeats from client, timeout: 5s (error level) |
| plain TCP connect and close, as a TCP health check does | nothing at all |
kill -9, a container stopped without a graceful shutdown. The duration in the line is how long the connection lived; a burst of these at deploy time points to shutdowns that do not close their connections.duration: connections that all die after the same idle time, here exactly 10 seconds.missed heartbeats from client. The client was alive but did not answer, typically because a long job blocked the thread that does the network I/O. pika reported StreamLostError: Transport indicated EOF after the job.Give every connection a name (connection_name in pika's client properties; most clients have a setting for it). The log line then shows which application it was, as billing-kill above.
connection.close(), so a deploy leaves no warnings and the ones that remain mean something.BlockingConnection call process_data_events while you wait, so heartbeats go out.