How can I create sidekiq queues with variable names at runtime?

Question

I am trying to create queues with variable queue names.

queue_name = "guide_" + guide['id'].to_s

Sidekiq::Client.push({
    'class' => GuidePdfWorker,
    'queue' => queue_name,
    'args'  => [key],
    'backtrace' => true
})

I know that I am supposed to add them to config/sidekiq.yml, but I can't, since I don't know the value of queue_name.

When I log Sidekiq::Client.registered_queues() I can see my queues, but they are never processed.

score 3 · Accepted Answer · answered Nov 22 '13 at 06:44

3

The Sidekiq Dynamic Queues gem will probably help you.

answered Nov 22 '13 at 06:44

davogones

7,321
31
36

1

Thanks :) I wanted to use it with [Sidekiq Limit Fetch](https://github.com/brainopia/sidekiq-limit_fetch) but I don't think that's possible. – Maxime Nov 30 '13 at 20:31
same problem here (http://stackoverflow.com/questions/20080047/how-to-push-job-in-specific-queue-and-limit-number-workers-with-sidekiq) – Matrix Mar 18 '15 at 14:38

score 0 · Answer 2 · answered Mar 26 '19 at 20:11

Just to provide a more complete, updated answer: there are plugins and extensions to Sidekiq that can do things like this, but Sidekiq is not designed to operate this way.

I don't recommend having more than a handful of queues ... and Sidekiq Pro cannot reliably handle multiple queues without polling

https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues

the number of named queues used should be minimized.

https://github.com/mperham/sidekiq/issues/835

Instead, consider having a known, static, queue with a worker that dispatches based on what you'd like to be dynamic:

class GuidePdfWorker
  include Sidekiq::Worker

  sidekiq_options queue: 'default'

  def perform(guide)
    # branch on guide['id'], perhaps re-queue in higher or lower priority queue?
  end
end

How can I create sidekiq queues with variable names at runtime?

2 Answers2

Linked