RabbitMQ: Are multiple consumers on one queue using a non-polling strategy possible?

Question

we use RabbitMQ to send jobs from a producer on one machine, to a small group of consumers distributed across several machines.

The producer generates jobs and places them on the queue, and the consumers check the queue every 10ms to see if there are any unclaimed jobs and fetch a job at a time if a job is available. If one particular worker takes too long to process a job (GC pauses or other transient issue), other consumers are free to remove jobs from the queue so that overall job throughput stays high.

When we originally set up this system, we were unable to figure out how to set up a subscriber relationship for more than one consumer on the queue that would prevent us from having to poll and introduce that little extra bit of latency.

Inspecting the documentation has not yielded satisfying answers. We are new to using message queues and it is possible that we don't know the words that accurately describe the above scenario. This is something like a blackboard system, but in this case the "specialists" are all identical and never consume each other's results -- results are always reported back to the job producer.

Any ideas?

Answer 1

Here you have to keep in mind that rabbitMQ channel is not thread safe. so create a singleton class that will handle all these rabbitmq operations

like

I am writing code sample in SCALA

    Object QueueManager{

      val FACTORY = new ConnectionFactory
      FACTORY setUsername (RABBITMQ_USERNAME)
      FACTORY setPassword (RABBITMQ_PASSWORD)
      FACTORY setVirtualHost (RABBITMQ_VIRTUALHOST)
      FACTORY setPort (RABBITMQ_PORT)
      FACTORY setHost (RABBITMQ_HOST)

    conn = FACTORY.newConnection
      var channel: com.rabbitmq.client.Channel =  conn.createChannel

    //here to decare consumer  for queue1
    channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
      channel.queueDeclare(QUEUE1, durable, false, false, null)
      channel queueBind (QUEUE1, EXCHANGE_NAME, QUEUE1_ROUTING_KEY)
      val queue1Consumer = new QueueingConsumer(channel)
      channel basicConsume (QUEUE1, false, queue1Consumer)

    //here to decare consumer  for queue2
    channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
      channel.queueDeclare(QUEUE2, durable, false, false, null)
      channel queueBind (QUEUE2, EXCHANGE_NAME, QUEUE2_ROUTING_KEY)
      val queue2Consumer = new QueueingConsumer(channel)
      channel basicConsume (QUEUE2, false, queue2Consumer)





    //here u should mantion distinct ROUTING key for each queue
       def addToQueueOne{
    channel.basicPublish(EXCHANGE_NAME, QUEUE1_ROUTING_KEY, MessageProperties.PERSISTENT_TEXT_PLAIN, obj.getBytes)
    }

   def addToQueueTwo{
channel.basicPublish(EXCHANGE_NAME, QUEUE2_ROUTING_KEY, MessageProperties.PERSISTENT_TEXT_PLAIN, obj.getBytes)
}

def getFromQueue1:Delivery={
 queue1Consumer.nextDelivery
}

def getFromQueue2:Delivery={
  queue2Consumer.nextDelivery
}

}

i have written a code sample for 2 queues u can add more queues like above........

Answer 2

Getting pub-subscribe is straight forward, i inital had same problems but works well. The project now has some great help pages at http://www.rabbitmq.com/getstarted.html

RabbitMQ has timeout and a resernt flag which can be used as you see fit.

You can also get the workers to be event driven as aposed to checking every 10ms etc. If you need help on this i have a small project at http://rabbitears.codeplex.com/ which might help slightly.