0

I would like to run multiple servers, each running a single Sidekiq process with multiple threads. My product issues several types of jobs, but one type of job often takes a long time and puts a heavy load on the server's CPU and memory.

When this happens, the job server goes down, making it impossible to run other types of jobs. Therefore, I would like to run high-load jobs and other jobs on separate servers while also improving job processing.

Is it possible to do this with Sidekiq?
If so, how can I allocate Sidekiq servers for each job?
I am using ActiveJob with Sidekiq, and both Rails and Sidekiq are up to date.

user20925957
  • 27
  • 4

1 Answers1

2

Yes, you can use host-specific queues, a technique documented on my blog. You create the job in a queue named after the hostname of the CPU-heavy machine:

queue_as 'big-dude.example.com'

these two are equivalent but set allows you to dynamically retarget the job from its default queue.

SomeJob.set(queue: 'big-dude.example.com').perform_later(...)

And then you'll start Sidekiq running on each machine, listening to its own host queue:

bundle exec sidekiq -q `hostname` -q default ...

Viola!

https://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/

Mike Perham
  • 21,300
  • 6
  • 59
  • 61