I have a collection of producers of jobs, and a (bigger) collection of workers that can process any job. Each job has a "reference", and jobs with the same reference MUST be processed sequentially, ie. only a single worker can ever be working on a job of a certain reference at any one time.
To guarantee fairness, it's also a requirement that a single "reference" must not be able to slow down the processing for other references.
Ref1 | Ref2 | Ref3 | ... | RefN
------|------|------|-----|------
a(1) | a(2) | | ... | a(3)
b(4) | b(5) | | ... | b(6)
c(7) | c(8) | | ... |
d(9) | | | ... |
e(10)| | | ... |
In this table, the columns contain the queued jobs for each reference, the letters are the ordering of the jobs within the reference, and the parenthesised numbers are the desired order in which jobs should be picked up by workers.
The references are dynamic and there might be thousands.
I checkout out beanstalkd, using tubes for the references, but this only guarantees sequentiality if each tube is watched by exactly one worker, which would mean I'd have to do a lot of bookkeeping (what if the assigned worker crashes? etc.)
Are there any job queue implementations that provide this functionality out of the box, or would I have to roll my own? I've got an implementation that does what I want, using a pretty naive sql table, but it performs terribly under load.
