Use bunny fetch for loading multiple messages from RabbitMQ

Question

I'm building a Ruby worker for RabbitMQ which will process multiple messages per time and will work on multiple queues. In RabbitMQ I have: Queue1, Queue2, Queue3...

I need to quickly load big pack of packets(with basic_get I can load 65535 messages without ask). Code below worked pretty well, until we moved to a new server. Latency between queue and worker go up, and I can load only 50 messages per seconds with basic get.

This is my workflow with get:

def work_iteration
  cache = queue_manager.messages
  size = cache.size
  return if size == 0
  delivery_tag = process(cache)
  queue_manager.ack(delivery_tag)
  size
end

and in QueueManager.rb I have

def messages
  MULTIPLE_MESSAGES.times.reduce([]) do |s, _|
    result = basic_get
    break s if result.nil? || result.last.nil?
    s << result
  end
end

def ack(delivery_tag)
  channel.ack(delivery_tag, true)
end

Another workaround I've found is to use subscribe method. But I haven't found way to break subscribe from inner loop.

queue_manager.subscribe do |delivery_info, properties, payload|
  cache << [delivery_info, properties, payload]
  if cache.size >= 65_000
    dt = process(cache)
    queue_manager.ack(dt)
  end
end

I googled and found that this can work for me:

length = queue.size
cache = [] 
consumer = queue.subscribe(block: false) do |delivery_info, properties, payload|
  cache << [delivery_info, properties, payload]
  if cache.size >= length
    dt = process(cache)
    queue.ack(dt)
    consumer.cancel
  end
end

But what happens in main thread then?
How to properly block main thread, and unblock when processing is finished?
Is there is other way to load multiple messages from queue, process them, close consumer and move to next queue?

Use bunny fetch for loading multiple messages from RabbitMQ

0 Answers0