6

In my application I have a queue that could potentially become very big. What if I discover that there's no more space on my machine? How can I split my queue on multiple machines? Maybe RabbitMQ philosophy is different and I should create multiple queues instead of one big queue..?

Best, Flavio

user2539645
  • 81
  • 1
  • 5

2 Answers2

2

As you can read on RabbitMQ mailing list thread http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-scalability-design-question-td28323.html, the solution I came up with is to implement the competing consumers pattern (many consumers on a queue) that when detects a special message (with stop flag on) sends a STOP message to a topic exchange.

This stop message is received by a "master" consumer for that queue that starts polling a Zookeeper (via curator) until all children of a ceratain zkNode have been deleted (using Zookpeer as a registry of queue-consumers in this case). When all consumers have finished their stopping phase the "master" consumer does some task and re-enable the original queue consumers sending a RESTART message to the topic exchange (where each consumer is listening to with a dedicated queue).

I hope this could help (or "inspire") someone else..

user2539645
  • 81
  • 1
  • 5
1

RabbimMQ provides clustering and high availability features right out of the box, you just have to configure them to your needs.

Actually, AMQP can hold messages of any size, but in most cases messages are just up to 32Kb in 99%, I guess. You can calculate estimate resource usage (min/max/avg) and make further decision do cluster or not to cluster.

pinepain
  • 12,453
  • 3
  • 60
  • 65
  • 2
    From what I understood queues cannot be splitted among different machines, a single queue resided only on a single node (so a single queue cannot scale so much)..am I wrong? – user2539645 Jul 19 '13 at 09:10
  • That's right, queue live on single node (but can be mirrored to another). What is your messages estimate size and flow rate? – pinepain Jul 19 '13 at 09:45
  • Actually is something that is going to be evaluated in the very next days..I expect terabytes of data coming in hours, so I should expect a very heavy loaded system..Maybe a better solution could be to create a new exchange instead of queue, and add more queue (demanding to the cluster) as the load grows..what do you think? The problem is: does the exchange create new queue evenly on the nodes (like round robin)? – user2539645 Jul 19 '13 at 09:53
  • In clustered env user can see all queues in it, so i guess broker free to pick any node to create queue. But AMQP is not aimed to _store_ terabytes of data but rather to deliver them. You can run numerous workers to process it. You can ask RabbitMQ guys on irc channel for details and post the answer here – pinepain Jul 19 '13 at 10:55