2

I just installed OpenStack Juno using devstack, and observed that RabbitMQ (package rabbitmq-server-3.1.5-10 installed by yum) is not stable, i.e. it quickly eats up the memory and shuts down; there is 2G of RAM. Below is the messages from logs and 'systemctl status' before the daemon died:

=INFO REPORT==== 18-Dec-2014::01:25:40 ===
vm_memory_high_watermark clear. Memory used:835116352 allowed:835212083

=WARNING REPORT==== 18-Dec-2014::01:25:40 ===
memory resource limit alarm cleared on node rabbit@node

=INFO REPORT==== 18-Dec-2014::01:25:40 ===
accepting AMQP connection <0.27011.5> (10.0.0.11:55198 -> 10.0.0.11:5672)

=INFO REPORT==== 18-Dec-2014::01:25:41 ===
vm_memory_high_watermark set. Memory used:850213192 allowed:835212083

=WARNING REPORT==== 18-Dec-2014::01:25:41 ===
memory resource limit alarm set on node rabbit@node.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************


rabbitmqctl[770]: ===========
rabbitmqctl[770]: nodes in question: [rabbit@node]
rabbitmqctl[770]: hosts, their running nodes and ports:
rabbitmqctl[770]: - node: [{rabbitmqctl770,40089}]
rabbitmqctl[770]: current node details:
rabbitmqctl[770]: - node name: rabbitmqctl770@node
rabbitmqctl[770]: - home dir: /var/lib/rabbitmq
rabbitmqctl[770]: - cookie hash: FftrRFUESg4RKWsyb1cPqw==
systemd[1]: rabbitmq-server.service: control process exited, code=exited status=2
systemd[1]: Unit rabbitmq-server.service entered failed state.

I know about set_vm_memory_high_watermark, but it doesn't solve the issue. I want to ensure that the daemon doesn't shut down abruptly. I wonder if someone saw this before and could advise?

Thanks.

UPDATE

Upgraded to version 3.4.2 taken directly from www.rabbitmq.com/download.html The new version doesn't consume RAM that fast and tends to work longer then previous version, but eventually still eats out all the memory and shuts.

Mark
  • 6,052
  • 8
  • 61
  • 129
  • The issue is most likely that one of the OpenStack services are not consuming messages properly and staking them up endlessly. Even on a big Openstack cluster the queues should rarely get bigger than 1-10 messages. – eandersson Dec 22 '14 at 13:54
  • @eandersson, thanks for comment. Can you tell how to check if a service processes/consumes messages appropriately? What log message pattern should I lookup in logs? – Mark Dec 22 '14 at 18:05
  • It's difficult to see using logs, besides by looking for obvious error messages in any of the Openstack service. The easiest would probably be to use the RabbitMQ Management page to look up which queue (belonging to which service) is queuing up the (most) messages. https://www.rabbitmq.com/management.html – eandersson Dec 22 '14 at 18:06
  • @eandersson, I enabled rabbitmq-management plugin, restarted rabbitmq and opened the management page at http://server-name:15672 then clicked to Queues tab, however it has 0 counters and doesn't show any unacked messages. Looks like it is operating normally. At the same time RabbitMQ daemon on the controller still increases the memory consumption, so I expect it will stop soon. – Mark Dec 24 '14 at 14:44
  • Honestly sounds like you don't have enough memory dedicated. Rabbit does not use a lot of memory in my experience, but if you are seeing a lot of traffic it will use its fair share of memory. I normally never run devstack with less than 4GB. – eandersson Jan 02 '15 at 19:36

2 Answers2

0

I think the number of connections in the servers are increasing and they are being held like that without closing that's why it is consuming more memory. When the usage of RAM increases beyond the watermark rabbitmq server won't accept any network request. Either you have to close the connections which all are opened or you have to increase the RAM of the system. But increasing the RAM will only reduce the problem for some time but you'll face the problem again it is better to close the connections.

Faiz Kidwai
  • 463
  • 5
  • 26
Rupendra
  • 1
  • 1
-1

try to use CloudAMQP instead of installing locally. This will be fixed then. firstly create a rabbitMQ account here. https://customer.cloudamqp.com/signup. then create your queue there and connect with your application.

dhanuka dulanjana
  • 1
  • 2