FTR, this was discovered in an escalation and the investigation is still in progress. We have a patch candidate but it needs testing. We also have a couple of potentially related issues which we haven't gotten to the bottom yet, which may or may not be actually related.

Here the steps to reproduce the issue:

1 - RabbitMQ cluster with 3 nodes

• Node 1 - 10.100.0.101
• Node 2 - 10.100.0.102
• Node 3 - 10.100.0.103

Setup this policy:

rabbitmqctl set_policy all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

2 - Execute the following script to one node, ex: Node 3

while true; do
echo "START"
sleep 5
rabbitmqctl list_channels
echo "sync 1 "
rabbitmqctl list_queues -q synchronised_slave_pids
sleep 2
rabbitmqctl stop_app
sleep 2
echo "stop 3"
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
sleep 3
rabbitmqctl start_app
sleep 5
echo "sync 2 "
rabbitmqctl list_queues -q synchronised_slave_pids

rabbitmqctl list_channels
iptables -A OUTPUT -d 10.100.0.101  -j DROP
echo "blocked"
sleep 25
iptables -F
echo "unblocked"
done

3 - execute this python or/and this java scripts

4 - after a few minutes you will see the command rabbitmqctl list_channels blocked.

On the management UI you will see channels with unknown connection

To make it clear: this issue is about (one more) missing/unreasonable timeout value. There can be more issues reproduced by the sequence posted by @Gsantomaggio. Our candidate patch introduces a timeout. The rest is still under investigation.

Does this bug affect RabbitMQ 3.6.1?

@falfaro this issue is solved in 3.6.1.

#675, referenced above, is a different issue which might be triggered in similar circumstances.