Resque: one worker per queue

Question

I currently have a Rails 3.0 project, with Ruby 1.9.2 and Resque.

My application has multiple worker classes and multiple queues, that are dynamically created (during runtime). Also, there are multiple workers started that are free to work on any queues, because at start time there isn't any existing queues, and they cannot be predicted:

$ COUNT=3 QUEUE=* rake resque:workers

Queues a created based on the project's id:

@queue = "project_#{project.id}".to_sym

For a given queue, their jobs have to processed in order and one at a time. My problem is that, by having multiple workers, multiple jobs are processed in parallel.

Is there a way to set the maximum number of workers per queue (to 1)? Is there a way to lock a queue while a job is processing?

Thanks!

score 2 · Answer 1 · answered May 29 '12 at 10:30

2

Resque-pool can help you specify number of workers per queue.

https://github.com/nevans/resque-pool

answered May 29 '12 at 10:30

Nick

518
1
4
16

Thanks Nick, but that doens't work for me, as my queues are created on runtime. – Henrique Zambon Jun 07 '12 at 20:35

score 2 · Accepted Answer · edited May 23 '17 at 12:14

2

I finally came to a quite simple solution using resque-retry and locks stored in redis (I am doing this for users, just do it for projects): https://stackoverflow.com/a/10933666/745266

edited May 23 '17 at 12:14

Community

1
1

answered Jun 07 '12 at 14:21

lacco

882
1
10
24

Looks good @lacco, thanks! Your problem was really similar as mine, so I'll definitely take a look at your solution soon. – Henrique Zambon Jun 07 '12 at 20:39

score 1 · Answer 3 · answered Feb 14 '12 at 18:16

1

The first solution I thought of would be to check if there is any worker working in a given queue when there's another worker polling that same queue. This could be done by reimplementing Resque::Job.reserve(queue):

module Resque
  class Job

    def self.reserve(queue)

      Resque::Worker.working.each do |worker|
        # if already working in the same queue
        if worker.job['queue'] == queue
          return
        end
      end

      return unless payload = Resque.pop(queue)
      new(queue, payload)
    end

  end
end

A possible issue would be the race condition. Thoughts?

answered Feb 14 '12 at 18:16

Henrique Zambon

1,301
1
11
20

Did you finally came to a solution, did the code work in production? – lacco Apr 04 '12 at 18:49
The solution turned out to be much more complicated than that and I decided to go with one worker only to guarantee my condition. Whenever performance becomes an issue, I will worry about it! :P – Henrique Zambon May 14 '12 at 11:03

Resque: one worker per queue

3 Answers3

Linked