RabbitMQ auto-configuration of cluster

Question

The RabbitMQ clustering document

https://www.rabbitmq.com/clustering.html

describes a process whereby you can deploy a configuration file for clustering so that a cluster is automatically created when your rabbitmq nodes boot.eg

[{rabbit,
  [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'], ram}}]}].

You apply this file on each of the 3 nodes at /etc/rabbitmq/rabbitmq.config, and then start rabbitmq on each and a cluster will apparently form.

This doesn't appear to work too well eg

If you start rabbit2, and rabbit3 hasn't already come up, the service will not start on rabbit2, as it is trying to create a cluster with rabbit3.

Equally, if you only apply the config on rabbit1, and have pre-started rabbit2 and rabbit3, rabbit1 will form a cluster with rabbit2 only, as, according to the documentation (I don't understand why):

RabbitMQ will try to cluster to each node provided, and stop after it can cluster with one of them.

The only way this seems to work is if you apply the config file on all 3 nodes, and have them start at exactly the same time. I am trying to deploy this with Ansible, which creates the nodes sequentially, so that doesn't work.

What am I missing here?

score 3 · Answer 1 · answered May 14 '15 at 21:44

I also deploy RabbitMQ clusters with Puppet, and you actually don't have to spin up all nodes at the exactly the same time.

What I usually do, and has so far worked for me is:

install RabbitMQ (RPM or DEB)
set up hosts file on each node, to conatins entries for all the three. example:

.

192.168.1.11    dev-c1n01-rabbitmq.example.com  dev-c1n01-rabbitmq
192.168.1.12    dev-c1n02-rabbitmq.example.com  dev-c1n02-rabbitmq
192.168.1.13    dev-c1n03-rabbitmq.example.com  dev-c1n03-rabbitmq

nodes we're clustering together (because I don't want to rely on DNS being available) * deploy rabbitmq.config

.

[
  {rabbit, [
    {cluster_nodes, {['rabbit@dev-c1n01-rabbitmq', 'rabbit@dev-c1n02-rabbitmq', 'rabbit@dev-c1n03-rabbitmq'], disc}},
    {cluster_partition_handling, pause_minority},
    {disk_free_limit, 2147483648},
    {heartbeat, 0},
    {tcp_listen_options, [binary, {backlog, 1024}, {nodelay, true}, {keepalive, true} ]},
    {vm_memory_high_watermark, 0.6},
    {default_user, <<"admin">>},
    {default_pass, <<"somedefaultpass">>}
  ]},
  {kernel, [

  ]}
,
  {rabbitmq_management, [
    {listener, [
      {port, 15672}
    ]}
  ]}
].
% EOF

deploy erlang.cookie

To create erlang cookie, I usually use http://passwordsgenerator.net/ and set it up to create 20 character string consistent only of uppercase characters. Then, put this string into /var/lib/rabbitmq/.erlang.cookie, like this:

echo -n 'LCQLSHVOPZFHRUXMMAPF' > /var/lib/rabbitmq/.erlang.cookie

start nodes (order doesn't matter as long as they have same erlang.cookie and rabbitmq.config)

This should work for you. Tested on 3.2, 3.3 and 3.5 versions.

I'm doing all of the same things with Ansible, but behaviour is erratic. On one run of the playbook, the cluster will form, but on the next, it won't, as one of the nodes in the proposed cluster fails because it can't find another. Its important to note that orchestration in Puppet and Ansible is different. In Puppet, orchestration is paralell, while in Ansible, it is sequential. — Garreth McDaid, May 15 '15 at 09:14
Thanks, that is the best answer. But let me add one more, you should reset each RabbitMQ server before creating a cluster as defined in [auto-config](https://www.rabbitmq.com/clustering.html#auto-config) section of rabbitmq documentation. — bhdrkn, Aug 26 '15 at 11:31

score 0 · Accepted Answer · answered May 15 '15 at 10:40

I've made some progress on this.

The issue appears to relate to the choice of using ram nodes rather than disc nodes in the rabbitmq.config file. From the documentation:

RAM nodes are an advanced use case; when setting up your first cluster you should simply not use them. You should have enough disc nodes to handle your redundancy requirements, then if necessary add additional RAM nodes for scale.

A cluster containing only RAM nodes is fragile; if the cluster stops you will not be able to start it again and will lose all data. RabbitMQ will prevent the creation of a RAM-node-only cluster in many situations, but it can't absolutely prevent it.

When I change the config file to use "disc" instead of "ram", cluster creation was a lot more stable.

[{rabbit,
    [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'],disc}}]}].

This is still not working for me .. If I use join_cluster, then it works but not with the above config. Anything else that I might be missing ? — Shrey, Dec 16 '15 at 06:37

RabbitMQ auto-configuration of cluster

2 Answers2