4

I have recently had to prioritize my Sidekiq queues (before I was using only the default queue).

So I have written few jobs which are supposed to be executed on the different queues.

My sidekiq.yml:

:queues:
  - slack_notifier
  - invoicing
  - default

deploy.rb (capistrano):

# ...
set :sidekiq_config, "#{current_path}/config/sidekiq.yml"
# probably below line does same thing as above one
set :sidekiq_options_per_process, ['--queue slack_notifier --queue invoicing --queue default']
# ...

And I can see while deploying, that it's being used:

01:32 sidekiq:start
      01 $HOME/.rbenv/bin/rbenv exec bundle exec sidekiq --index 0 --pidfile /home/ubuntu/cap/shared/tmp/pids/sidekiq.pid --environment production --logfile /home/ubuntu/cap/shared/log/sidekiq.log --config /home/ubuntu/cap/current/config/sidekiq.yml --queue slack_notifier --queue invoicing --queue default --daemon

Sidekiq is running:

ps aux | grep '[s]idekiq'
#=> ubuntu     716 16.4  2.1 668604 176860 ? Sl   07:31  14:18 sidekiq 4.2.6 cap [1 of 1 busy]

Now, when I start off the jobs, they just seem to be halted at some point.

Not sure if I can describe the issue better. When I clear out the two priority queues (slack_notifier and invoicing) the default still does not seem to process jobs.

Andrey Deineko
  • 51,333
  • 10
  • 112
  • 145
  • 6
    Yep, downvotes is what I'm here for. – Andrey Deineko Dec 19 '17 at 09:04
  • 4
    I am unsure why this question was downvoted; upvoted for the sake of entropy constancy. – Aleksei Matiushkin Dec 19 '17 at 10:22
  • It seems you have a lot of jobs/long job and Sidekiq with concurrency=1. You can try to check what is going on with rails console and https://github.com/mperham/sidekiq/wiki/API – Pavel Mikhailyuk Dec 19 '17 at 10:25
  • Have you tried three different sidekiq runners, one per queue, as [described here](https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues)? – Aleksei Matiushkin Dec 19 '17 at 10:26
  • Also, increase a concurrency to be at least `10`. – Aleksei Matiushkin Dec 19 '17 at 10:27
  • @mudasobwa it is working locally when I do `bundle exec sidekiq --config config/sidekiq.yml` - jobs on all three queues are being executed. What I initially wanted is actually having those reserved queues you mentioned, but I do not know how to set it up with capistrano's `deploy.rb` – Andrey Deineko Dec 19 '17 at 10:37
  • 2
    That’s not a duty of capistrano: you deploy it once, you start it three times. Just create three `upstart`/`systemctl` configs for them. – Aleksei Matiushkin Dec 19 '17 at 10:42
  • @mudasobwa true, worked. Please, make your suggestion an answer if you will, I'll accept it. Thanks for hints! – Andrey Deineko Dec 20 '17 at 07:15
  • @AndreyDeineko I believe you’d better describe the winning path that is proven to work. This could be a good reference in the future for “sidekiq queue problem” landing, so it makes sense to provide a good answer that works rather than a good hint that helped :) – Aleksei Matiushkin Dec 20 '17 at 07:26
  • @mudasobwa damn man how are you so right every single time? :) I'll make sure to describe the solution later (I still have to set up the `systemctl` config - ATM I just have tested it with manually firing 3 queues on prod server). – Andrey Deineko Dec 20 '17 at 07:36
  • Also the way you define your queues implies all jobs in slack_notifier get processed first, then invoicing and last the jobs in default. That might be the reason for delay. You can use [slack_notifier, 2] notation instead where the second number is the weight which tells sidekiq that this queue must be checked twice as often (in case default was [default, 1]). – amit_saxena Apr 03 '19 at 14:59

1 Answers1

0

Maybe you can run multiple processes for all queues, put separate queues in separate processes for processing sidekiq jobs. Your configuration will look like this

set :sidekiq_processes, 3
set :sidekiq_options_per_process, ['--queue slack_notifier', '--queue invoicing', '--queue default']

I think it will not halt the sidekiq, but the precondition to everything is your sidekiq processes are optimized enough so they are not taking more memory than they are supposed to be using.

usama
  • 78
  • 12