0

I'm having an application with a couple thousand users. Each user can have 10 - 100's of jobs in the queue. I wan't my workers (~10) to process 10 jobs of 10 different users instead of just 10 jobs (which could be from the same user).

user1
    job1
    job2
    job3
    job4
    job5
user2
    job6
    job7
    job8
user3
    job9
    job10
    ...

So in the above example I want my workers to process in the following order:

worker1 -> job1, job2, job3
worker2 -> job6, ...
worker3 -> job9, ...
...

Is there any way this is easy to implement with beanstalkd (prefered) or gearman?

Floris
  • 299
  • 3
  • 17
  • Could you be more precise about your constraint ? If you know how to do to execute 10 jobs for 1 user, why did you blocked when you want to execute 10 jobs for 10 user ? – FArcellier Nov 19 '13 at 08:19
  • I've changed my original post so it's a bit more detailed. – Floris Nov 19 '13 at 18:28

1 Answers1

0

In this case I think you can use tube feature.

you can have
  • tube for each user(user1, user2, user3,...)
  • controller tube
then step will be like this

job producer:

  1. add job to user's tube.
  2. add tube's name that you just inserted to controller tube.

job worker:

  1. reserve(with no timeout) to controller tube
  2. got job from controller tube
  3. loop reserve-with-timeout(with timeout = 0) to user's tube until you got TIMED_OUT
  4. start again with 1.

reverse This will return a newly-reserved job. If no job is available to be reserved, beanstalkd will wait to send a response until one becomes available.

reverse-with-timeout A timeout value of 0 will cause the server to immediately return either a response or TIMED_OUT. A positive value of timeout will limit the amount of time the client will block on the reserve request until a job becomes available.

reference docs

zinuzoid
  • 337
  • 8
  • 14
  • This way I would get easily a couple thousand tubes as the user amount could be from 500 up to over 10k! – Floris Jan 23 '14 at 09:59