1

We have currently using a service bus in Azure and for various reasons, we are switching to RabbitMQ. Under heavy load, and when specific tasks on backend are having problem, one of our queues can have up to 1 million messages waiting to be processed.

RabbitMQ can have a maximum of 50 000 messages per queue. The question is how can we design the rabbitMQ infrastructure to continue to work when messages are temporarily accumulating?

Note: we want to host our RabbitMQ server in a docker image inside a kubernetes cluster. we imagine an exchange that would load balance mesages between queues in nodes behind. But what is unclear to us is how to dynamically add new queues on demand if we detect that queues are getting full.

CloudAnywhere
  • 669
  • 1
  • 11
  • 27

2 Answers2

3

RabbitMQ can have a maximum of 50 000 messages per queue.

There is no this kind of limit. RabbitMQ can handle more messages using quorum or classic queues with lazy. With stream queues RabbitMQ can handle Millions of messages per second.

we imagine an exchange that would load balance messages between queues in nodes behind.

you can do that using different bindings.

kubernetes cluster.

I would suggest to use the k8s Operator

But what is unclear to us is how to dynamically add new queues on demand if we detect that queues are getting full.

There is no concept of FULL in RabbitMQ. There are limits that you can put using max-length or TTL.

Gabriele Santomaggio
  • 21,656
  • 4
  • 52
  • 52
  • 1
    yes, tested. I could create 1 Million messages in a queue. The 50 000 messages limlit came from a misleading information from https://www.cloudamqp.com/blog/part1-rabbitmq-best-practice.html – CloudAnywhere Apr 23 '22 at 16:52
  • I will investigate the streams option. What I like is the persistence and the fact that messages will not be lost if the RabbitMQ server is restarted by k8s. – CloudAnywhere Apr 23 '22 at 16:53
0

A RabbitMQ queue will never be "full" (no such limitation exists in the software). A queue's maximum length rather depends on:

  1. Queue settings (e.g max-length/max-length-bytes)
  2. Message expiration settings such as x-message-ttl
  3. Underlying hardware & cluster setup (available RAM and disk space).

Unless you are using Streams (new feature in v 3.9) you should always try to keep your queues short (if possible). The entire idea of a Message Queue (in it's classical sense) is that a message should be passed along as soon as possible.

Therefore, if you find yourself with long queues you should rather try to match the load of your producers by adding more consumers.

MrD
  • 819
  • 6
  • 6