17

I'd like to setup sidekiq so that it runs on a separate host from my rails app, to avoid weighing down the web server when running memory-intensive background tasks.

What are the configuration details I need to look at?

Are there known best practices or configurations for this?

EDIT:

To clarify, what I mean to ask is, how do I configure rails so that sidekiq API calls (MyWorker.perform_async) would run if the sidekiq process isn't running locally?

Ovesh
  • 5,209
  • 11
  • 53
  • 73
  • Have you gone through [this wiki on Sidekiq's Github page](https://github.com/mperham/sidekiq/wiki/Advanced-Options)? – Kashyap Aug 16 '13 at 04:13
  • Yes, but I couldn't find anything related to what I'm asking. Please correct me if I'm wrong. – Ovesh Aug 16 '13 at 04:46

1 Answers1

24

Edit to clarify

Before everything else, you should understand that when you run perform_async, it simply sticks the Jobs into a Redis queue. Sidekiq watches this queue and runs jobs when they come in. So as long as your app and your Sidekiq are looking at the same Redis server (and same database, of course), things will work.

Original answer

First, as you can probably guess, it will need a duplicate checkout of your code so that it can run. No big deal.

Secondly, it will need access to your database and redis servers on the other box. That means making sure those ports are open on the other server. That can get tricky, b/c you ideally don't want those exposed to the open Internet. Usually, for a multi-box setup, you'll have only one box exposed to the open Internet. It communicates with the rest of your boxes over private IPs:

Public Web server

Runs Apache/Nginx and maybe your app servers.

Private App Server(s) (optional)

Runs your app servers, if they aren't running on the Public server. Connects to your Database and Redis server.

Private Sidekiq Server(s) (optional)

Runs Sidekiq. Connects to your Database and Redis server.

Private Database/Redis Server

Runs database and Redis. Of course they can split out to different servers if required.

bioneuralnet
  • 5,283
  • 1
  • 24
  • 30
  • I guess my question was, when you call `MyWorker.perform_async`, how would it work if sidekiq isn't running locally? How is that configured? – Ovesh Aug 16 '13 at 04:44
  • It's configured as I stated above. Point Sidekiq to whatever host Redis is running on (instead of "localhost:6379"). There's really nothing different about it. – bioneuralnet Aug 16 '13 at 16:17
  • 2
    Said another way: Instead of booting Sidekiq locally you've created another host, installed your Rails app, pointed it's DB and Redis config to your original server, then booted up Sidekiq. Done, that's all that's changed. – bioneuralnet Aug 16 '13 at 16:31
  • 1
    That last comment made it click in my head. So you're saying, under the hood, When I call `MyWorker.perform_async`, what actually happens is an entry gets added to redis, and sidekiq subscribes to redis events? – Ovesh Aug 17 '13 at 13:31
  • Precisely. Sorry, I should have stated that earlier. – bioneuralnet Aug 19 '13 at 21:49
  • Thank you, great answer. How about updating the answer body to reflect this explanation? – Ovesh Aug 20 '13 at 06:42
  • It will be great If it has a solid example. – new2cpp Apr 25 '17 at 14:35