A queuing system which supports job batching (e.g. several jobs for 1 worker at once)

Question

I'm looking for a queuing system that could support the following scenario:

A client adds a job - to check how many Facebook likes a particular url (URL1) has;
A client adds another job - to check the same information for URL2;
[....]
A worker picks up anything from 1 to 50 jobs (urls) from the queue (e.g., if there's only 5 - it picks up 5, if there's 60 - picks up 50, leaving others for another worker), and issues a request against Facebook API (which allows multiple urls per request). If it succeeds, all jobs are taken out of the queue, if it fails - all of them stay.

I'm working with PHP and I've looked into Gearman, Beanstalkd, but did not find any similar functionality. Is there any (free) queuing system that would support such a "batch-dequeuing"?

Or, maybe, anybody could suggest an alternative way of dealing with such an issue? I've considered keeping a list of "to check" urls outside the queuing system and then adding them in bundles of max N items with a cron job that runs every X period. But that's kind of building your own queue, which defeats the whole purpose, doesn't it?

It's not the cleanest way to do it, but I think that with Beanstalk you can have one worker try to get multiple jobs (just use a loop). — trutheality, Mar 03 '12 at 23:10
@trutheality I'll definitely take a look - so far could not find anybody mentioning such a possibility, though :/ — Aurimas, Mar 04 '12 at 21:28

score 1 · Accepted Answer · answered Mar 29 '12 at 21:45

I've used Beanstalkd to fetch 100 twitter names at a time, and then calling an API with them all. When I was done, I deleted them - but I could have elected to not delete some (or all) if I wished.

It was a simple loop to reserve the initial 100 (one at a time), and I put the results (the job ID and the data returned) into an array. When I was done dealing with the payload (in this instance, a twitter screen-name), I went through deleting them - but I could have easily have released them back into the queue.

Thanks - I see how you can do it with Beanstalkd. I went with Gearman in the end, so now I just need to figure out if I can do the same with Gearman. — Aurimas, Mar 30 '12 at 11:29

score 0 · Answer 2 · answered Mar 03 '12 at 22:59

0

Perhaps you could take inspiration from MediaWiki's job queue system. Not very complicated, but it does have some issues that you may run into if you decide to roll your own.

The DB tables used for this are defined here.

answered Mar 03 '12 at 22:59

Etienne Perot

4,764
7
40
50

Thanks for the suggestion, but I don't really think that this queue system supports what I am looking for, either (unless I missed it). It's a good starting point for developing own queue system, nevertheless! – Aurimas Mar 04 '12 at 21:22

A queuing system which supports job batching (e.g. several jobs for 1 worker at once)

2 Answers2

Linked