24

I'm not familiar with the internals of Sidekiq and am wondering if it's okay to launch several Sidekiq instances with the same configuration (processing the same queues).

Is it possible that 2 or more Sidekiq instances will process the same message from a queue?

UPDATE:
I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine

Thilo
  • 17,565
  • 5
  • 68
  • 84
Jacob
  • 1,642
  • 2
  • 15
  • 27
  • Why would you want to do that? The advantage of sidekiq over resque is supposed to be multi-threading, so only one process is needed. – Thilo Nov 03 '12 at 22:04
  • 2
    I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine. Its not a question about threading vs. processes. – Jacob Nov 03 '12 at 22:31
  • 5
    @Thilo, even on a single machine, while "only one process is needed" that limits the cpu use to 100% of a single core, which limits the speed of processing the queue. I have an 8-core machine and want siekiq to use to use 4 cores at full capacity, so I need 4 processes. – Ben Lee May 13 '13 at 19:09

2 Answers2

48

Yes, sidekiq can absolutely run many processes against the same queue. Redis will just give the message to a random process.

Mike Perham
  • 21,300
  • 6
  • 59
  • 61
  • Can issues arise if different instances run the same queries at very similar times (or does that make any sense), for example with an accidental double form submission? – atw Jan 15 '18 at 14:25
  • 1
    Bugs can happen but Sidekiq is designed to be concurrent. The application code is much more likely to be buggy with a double click. – Mike Perham Jan 15 '18 at 16:11
  • Random is effectively even? Is that the thinking? – jprio May 24 '23 at 22:01
11

Nope, I've ran Sidekiqs in different machines with no issues.

Each of the Sidekiqs read from the same redis server, and redis is very robust in multi-threaded, and distributed scenarios.

In addition, if you look at the web interface for Sidekiq, it will show all the workers across all machines because all the workers are logged in the same redis server.

So no, no issues.

Henley
  • 21,258
  • 32
  • 119
  • 207
  • 3
    this is interesting. I have several Sidekiq servers running from different machines, and even though they consume the jobs correctly from the Redis queue, I don't see the processes that are in a different machine than my Rails app showing up in the UI. Did you have to set any special config to do so? – Nobita Jul 03 '14 at 16:36
  • I have 2 instances and the same code is running on that 2 instances. Both instances have sidekiq running and sharing common redis. Is it possible that the job scheduled from instance-1 will execute in instance-2? If so then what are the possible way to prevent this? – Sanjay Prajapati Oct 12 '21 at 07:36