31

I'm trying to run RabbitMQ on a small VPS (512mb RAM) along with Nginx and a few other programs. I've been able to tweak the memory usage of everything else without difficulty, but I can't seem to get RabbitMQ to use any less RAM.

I think I need to reduce the number of threads Erlang uses for RabbitMQ, but I've not been able to get it to work. I've also tried setting the vm_memory_high_watermark to a few different values below the default (of 40%), even as low as 5%.

Part of the problem might be that the VPS provider (MediaTemple) allows me to go over my allocated memory, so when using free or top, it shows that the server has around 900mb.

Any suggestions to reduce memory usage by RabbitMQ, or limit the number of threads that Erlang will create? I believe Erlang is using 30 threads, based on the -A30 flag that I've seen on the process command.

Ideally I'd like RabbitMQ mem usage to be below 100mb.

Edit:

With vm_memory_high_watermark set to 5% (or 0.05 in the config file), the RabbitMQ logs report that RabbitMQ's memory limit is set to 51mb. I'm not sure where 51mb is coming from. Current VPS allocated memory is 924mb, so 5% of that should be around 46mb.

According to htop/free before starting up RabbitMQ, I'm sitting around 453mb of used ram, and after start RabbitMQ I'm around 650mb. Nearly 200mb increase. Could it be that 200mb is the lower limit that RabbitMQ will run with?

Edit 2

Here are some screenshots of ps aux and free before and after starting RabbitMQ and a graph showing the memory spike when RabbitMQ is started.

Edit 3

I also checked with no plugins enabled, and it made very little difference. It seems the plugins I had (management and its prerequisites) only added about 8mb of ram usage.

Edit 4

I no longer have this server to test with, however, there is a conf setting delegate_count that is set to a default of 16. As far as I know, this spawns 16 sup-procs for rabbitmq. Lowering this number on smaller servers may help reduce the memory footprint. No idea if this actually works, or how it impacts performance, but it's something to try.

Alex Jillard
  • 2,792
  • 2
  • 19
  • 20

2 Answers2

11

The appropriate way to limit memory usage in RabbitMQ is using the vm_memory_high_watermark. You said:

I've also tried setting the vm_memory_high_watermark to a few different values below the default (of 40%), even as low as 5%.

This should work, but it might not be behaving the way you expect. In the logs, you'll find a line that tells you what the absolute memory limit is, something like this:

=INFO REPORT==== 29-Oct-2009::15:43:27 ===
Memory limit set to 2048MB.

You need to tweak the memory limit as needed - Rabbit might be seeing your system as having a lot more RAM than you think it has if you're running on a VPS environment.

Sometimes, Rabbit can't tell what system you're on and uses 1GB as the base point (so you get a limit of 410MB by default).

Also, make sure you are running on a version of RabbitMQ that supports the vm_memory_high_watermark setting - ideally you should run with the latest stable release.

Rob Harrop
  • 3,445
  • 26
  • 26
  • When I used `vm_memory_high_watermark` it reported the correct ram limit in the log (based on the VPS's actual 900mb, not the 512mb I purchased). When set to 5% the logs showed 45mb, but each erlang process that spawned used between 20 and 25mb of ram, which quickly put me over my 512mb allotment. – Alex Jillard Jun 28 '11 at 20:25
  • When you say 'each erlang process' do you mean, OS erlang process or erlang internal process? – Rob Harrop Jun 28 '11 at 20:26
  • When I run htop, I see 30 or so `/usr/lib/erlang/erts-5.7.4/bin/beam.smp -W w -K true -A30 -P 1048576` and then some flags related to RabbitMQ and mnesia. – Alex Jillard Jun 28 '11 at 20:35
  • That is **not** good. RabbitMQ should only be running as one OS process. Is it possible that you've got a few instances left over from testing. Try `killall beam.smp` to get rid of them. I just verified that I can run a single instance of RabbitMQ in 100MB on my machine. – Rob Harrop Jun 28 '11 at 20:49
  • 3
    When I stop rabbitmq (/etc/init.d/rabbitmq-server stop) all the processes go away. When running `top` it shows the single rabbitmq-server process and the single beam.smp process. However, when I run `htop` I see all of the beam.smp procs. After I `killall beam.smp` the rabbitmq-server process is gone. Once starting it back up again, I get the same results as before. – Alex Jillard Jun 28 '11 at 20:55
  • What does `top` say about memory usage? I'm guessing you've got `htop` configured to show userland threads, you don't actually have multiple processes running - the lines are just duplicates per thread - `top` will give you definitive memory usage for the entire process. – Rob Harrop Jun 28 '11 at 21:05
  • top shows that beam.smp is using 35m (3.9%) and VIRT is 402m. I don't think this explains the 200mb increase in ram usage after starting RabbitMQ, though. – Alex Jillard Jun 30 '11 at 04:45
  • There should be no new processes other than the beam.smp process running after starting RabbitMQ. Can you snapshot the output of `ps` before and after starting to get a diff? Might show up something unexpected. – Rob Harrop Jul 01 '11 at 14:24
  • Edited the question to include before/after screenshots for `ps`/`free`. To keep it as clear as possible, I stopped all other processes. – Alex Jillard Jul 02 '11 at 16:26
1

Make sure to set an appropriate QoS prefetch value. By default, if there's a client, the Rabbit server will send any messages it has for that client's queue to the client. This results in extensive memory usage both on the client & the server.

Drop the prefetch limit down to something reasonable, like say 100, and Rabbit will keep the remaining messages on disk on the server until the client is really ready to process them, and your memory usage will go way way down on both the client & the server.

Note that the suggestion of 100 is just a reasonable place to start - it sure beats infinity. To really optimize that number, you'll want to take into consideration the messages/sec your client is able to process, the latency of your network, and also how large each of your messages is on average.

metaforge
  • 911
  • 10
  • 11