A queueing system for Perl

Question

I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs.

I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so jobs enqueued. What are good alternatives to this module which can be used in Perl?

score 9 · Accepted Answer · answered Apr 13 '12 at 14:51

9

I've had pretty good success with using ZeroMQ for this sort of problem, both with Perl and other languages.

In my experience, the ZeroMQ module appears to be the most reliable binding for Perl currently.

answered Apr 13 '12 at 14:51

rafl

11,980
2
55
77

ZeroMQ looks very interesting, I'll try it. – Eugene Yarmash Apr 13 '12 at 18:51

score 4 · Answer 2 · answered Apr 13 '12 at 16:03

I haven't tried it under the conditions you list, but Thread::Queue has proven useful to me. Combined with forks, it can be used to communicate with processes, as long as those processes were spawned by the queue creator.

use forks;  # If you want to use processes instead of threads.
use Thread::Queue qw( );

A worker model is usually ideal.

my $q = Thread::Queue->new();

my @workers;
for (1..$NUM_WORKERS) {
   push @workers, async {
      while (my $item = $q->dequeue()) {
         ...
      }
   };
}

# ... Enqueue requests [ $q->enqueue($request); ] ...

# Signal termination
$q->enqueue(undef) for 1..@workers;

# Collect workers.
$_->join() for @workers;

A queueing system for Perl

2 Answers2

Linked