0

I've been trying to use RabbitMQ perl library Net::RabbitFoot which uses AnyEvent::RabbitMQ underneath. According to RabbitMQ Tutorial, setting prefetch_count to 1 should ensure fair dispatch, as in should not dispatch a message to a worker that is already busy on another message. However, the perl implementation Net::RabbitFoot, does not seem to work that way even after setting the qos as described here, line 54. It seems to just do vanilla round-robin dispatch and ends up dispatching to machine that is already executing a job. This is the qos implementation. Could you help me with figuring out why this is happening? Is it a bug in the library?

Thanks in advance.

Edit:

This is my setup: 2 consumers attached to the same-named queue. When I dispatch a lot of messages, I see this pattern: Consumer 1: Msg1, Msg3, Msg5 ... Consumer 2: Msg2, Msg4, ... All messages are from the same queue. What happens now is if Msg3 hogs Consumer 1, still Msg5 is sent to Consumer 1 while Consumer 2 is sitting free.

alpha_cod
  • 1,933
  • 5
  • 25
  • 43

1 Answers1

2

vanilla round-robin? uh?

The prefetch_count=1 comes useful when there are many consumers attached to the same common queue. In fact by default the client libraries will prefetch many messages in one shot.

So the default odd effect, that you want to avoid by setting it to one, is that one client get most (or all) the messages, and other consumers get few or none, being the load unbalanced.

However you speak of "vanilla round-robin": that happens when you have different (probably unnamed/temporary) queues attached to a direct exchange, one per consumer. But in this way you have no way to balance the load dynamically.

If I'm guessing right you need to change your configuration and let all the consumers attach to the same named queue.

EDIT: from the comment of the OP, this is not the case.

Alternatively it's possible that your consumers are configured with auto-ack, or they do send the ACK before completing their job. In this case too the RabbitMQ client API thinks that it's free to get another message: you need to send the ack back only after the local task regarding that message has been completed.

Sigi
  • 4,826
  • 1
  • 19
  • 23
  • This is my setup: 2 consumers attached to the same-named queue. When I dispatch a lot of messages, I see this pattern: Consumer 1: Msg1, Msg3, Msg5 ... Consumer 2: Msg2, Msg4, ... All messages are from the same queue. What happens now is if Msg3 hogs Consumer 1, still Msg5 is sent to Consumer 1 while Consumer 2 is sitting free. This is the problem I'm trying to address – alpha_cod Apr 02 '14 at 22:18
  • added an alternative for this behavior to the answer - see if it matches! – Sigi Apr 03 '14 at 07:48
  • I just checked my consumer, it sends an ack at the end of its consume callback method after completing its job like this $channei->consume( on_consume => sub { my $var = shift; ..work.. $channel->ack(); } ); Apart from this should I make sure its not auto configured to do an auto-ack while opening the channel by setting some configuration parameter? – alpha_cod Apr 03 '14 at 21:45
  • The declare_queue method in this http://cpansearch.perl.org/src/BOBTFISH/AnyEvent-RabbitMQ-1.08/lib/AnyEvent/RabbitMQ/Channel.pm is setting no_ack to 1, does that mean the broker doesnt wait for an ack from the consumer before marking it as available again? – alpha_cod Apr 03 '14 at 21:59
  • 1
    yes! That seems your problem actually. you need to set no_ack=false (0) to let ack's to be considered at all. no_ack=1 means "don't wait for the ack's". – Sigi Apr 03 '14 at 22:18