2

I have built an application which consists of one publisher, several queues and several consumers for each queue. Consumers on a queue (including the queue) share the channel. Other queues use a different channel. I am observing that for different queues, tasks are being worked on parallel but for a specific queue this is not happening. If I publish several messages at once to a specific queue, only one consumer works while the other ones wait until the work is ended. What should I do in order for consumers to work on parallel? 

workers.each do |worker|
  worker.on_delivery() do |delivery_info, metadata, payload|
    perform_work(delivery_info, metadata, payload)
  end
  queue.subscribe_with(worker)
end

This is how I register all the consumers for a specific queue. The operation perform_work(_,_,_) is rather expensive and takes several seconds to complete.

Yeezus
  • 95
  • 9

1 Answers1

3

RabbitMQ works off the back of the concept of channels, and channels are generally intended to not be shared between threads. Moreover, channels by default have a work thread pool size of one. A channel is an analog to a session.

In your case, you have multiple consumers sharing a queue and channel, and performing a long-duration job within the event handler for the channel.

There are two ways to work around this:

  1. Allocate a channel per consumer, or
  2. Set the work pool size of the channel on creation See this documentation

I would advocate 1 channel per consumer since it has a lower chance of causing unintended side-effects.

mcfinnigan
  • 11,442
  • 35
  • 28
  • Thanks for the answer! I tried the first method and it works as expected. Would you still advocate for 1 channel per consumer if the number of consumers is "huge"? – Yeezus Oct 23 '18 at 11:51
  • Yes. channels are lightweight and rabbitmq is very memory efficient. You need to read up on the implications of concurrent message delivery and acknowledgement modes and be sure you understand the implications before doing multiplexing over a channel. 1 channel per consumer is simpler and while more resource intensive it compartmentalises workers from eachother. – mcfinnigan Oct 23 '18 at 12:13