RabbitMQ - unexpected empty and unsynchronized queues after cluster node failure

Question

I have downloaded and installed RabbitMQ in cluster on Windows:

• downloaded and installed ERLANG and RABBITMQ on two servers (for example Alpha and Bravo)
• installed management plugin, set custom tcp_listener port (to increase security) and add custom admin login instead of default guest:guest login
• set same cookie on both machines (in both folders C:\Windows\ and C:\Users\username)
• installed RabbitMQ as Windows Service
• joined Bravo node to Alpha node to cluster
• added policy to mirror both all queues (Pattern="", Definition="ha-mode: all")
• I created durable mirrored queue and publish message to it - it is synchronized between both nodes (it works fine as I expected)

But then I tried to test what will happen if I turn off one node:

• I turned off node Bravo
• I added message to Alpha node
• I turned back on node Bravo
• missing message was synchronized to Bravo node but in management console this node was marked as "unsynchronized"

What is wrong this queue? But there are other problems. For example:

• I turned off node Bravo
• I added message to Alpha node
• I turned back on node Bravo and waited to synchronize message
• I turned off Alpha node
• I added message to Bravo node
• I turned Alpha node back on
• all added messages in queue magically disappeared

This is really wierd. Anybody has idea what happened?

Also there is third test case:

• I have turned off Alpha node, added message to Bravo and turn also Bravo node off
• then I turned on Alpha node and I recieved message "timeout_waiting_for_tables". I tried it more times. It started to work only when I turned also Alpha node back on

Maybe I just didn't get how clustering works in RabbitMQ. Can anyone help me and tell what is going on?

Answer 1

Same problem but there is a little understanding to be done as well as a possible pitfall.

Firstly, I was fooled by the fact that I didn't pass my vhost to the command:

rabbitmqctl set_policy -p myvhost HA '*' '{"ha-mode": "all"}'

Otherwise the vhost defaults to "/"

After this, when I logged onto the web console, I saw that the node field was reporting on two nodes ...now. Great :-)

However, if you bring one up and down, then the other up and down, the queue disappears!? This is because there is NO "synchronisation" in the mirroring, ONLY "stacking". Meaning if you bring a node down, the rest of the messages are served from the remaining node (or nodes). If you bring a new/existing node up, it will only mirror NEW messages that are added.

I'm fairly new to this so I would assume that having 3 nodes would be far better than two. This means that if one node goes down, there is still resiliance over the other two nodes (depending what your biz case is right). Of course if two nodes go down, you have lost replication for anything left in the queues. I reckon this should be called the "3 strike setup"!