11

I have a rake task which is going to call 4 more rake tasks, in order:

rake:one
rake:two
rake:three
rake:four

Rake tasks one, two, and three are getting data and adding it to my database. Then rake:four is going to do something with that data. But I need to make sure that one, two, and three are complete first. Each rake task is actually spinning up Sidekiq workers to run in the background. In this scenario, would all of the workers created by rake:one finish first, then rake:two, etc?

If not, how can I ensure that the workers are executed in order?

Luigi
  • 5,443
  • 15
  • 54
  • 108

2 Answers2

9

Sidekiq processes jobs in the order which they are created, but by default it processes multiple jobs simultaneously, and there is no guarantee that a given job will finish before another job is started.

Quoting from https://github.com/mperham/sidekiq/wiki/FAQ:

How can I process a certain queue in serial?

You can't, by design. Sidekiq is designed for asynchronous processing of jobs that can be completed in isolation and independent of each other. Jobs will be popped off of Redis in the order in which they were pushed but there's no guarantee that Job #1 will execute fully before Job #2 is started.

If you need serial execution, you should look into other systems which give those types of guarantees.

Note you can create a Sidekiq process dedicated to processing a queue with a single worker. This will give you serial execution but it's a hack.

Also note you can use third-party extensions for sidekiq to achieve that goal.

Nathan
  • 7,816
  • 8
  • 33
  • 44
0

You can simply create one meta rake task, which will include all those tasks in right order.

Or as a less hacky solution: reduce number of workers per queue to 1: https://github.com/brainopia/sidekiq-limit_fetch#limits

And add all your jobs to this queue

Ivan Denysov
  • 1,502
  • 12
  • 16
  • That's what I've done. But each rake task creates 1000 workers then goes to the next one. Essentially I have 4000 workers created in about 5 seconds, it doesn't take long for the rake task to run. So, will those 4000 workers be executed in the order they were created? – Luigi Mar 24 '14 at 15:49
  • @Luigi given there is only ever a *SINGLE* worker, they will be executed in the order they were enqueued. – Michael Jan 25 '18 at 23:38