5

Given that unicorn usually manages more than one Rails server process, and given that a Resque job runner probably consumes less resources than a Web request, it should be possible to run more than one resque worker on a single Heroku dyno.

Is anyone doing this successfully so far? My thoughts are, that an easy way to do so would have the Procfile runs foreman, which then runs 2 (or more) instances of the actual worker (i.e. rake resque:work)

Or is rake resque:workers up to that task? Resque itself does not recommend using that method, as this starts workers in parallel threads instead of in parallel processes.

Obviously, this makes sense only on i/o bound jobs.

radiospiel
  • 2,450
  • 21
  • 28

2 Answers2

8

One can use foreman to start multiple processes. Add foreman to your Gemfile, and then create two files:

Procfile:

worker: bundle exec foreman start -f Procfile.workers

Procfile.workers:

worker_1: QUEUE=* bundle exec rake resque:work
worker_2: QUEUE=* bundle exec rake resque:work

The same technique can be used to run a web server alongside some workers.

NOTE: while many state success using this approach, I would not suggest to use it outside of some experiments, mostly because of the risk to run into RAM limitations on small heroku instances; and once you pay for the heroku service it is probably easier to just spin up a dedicated worker machine anyways.

radiospiel
  • 2,450
  • 21
  • 28
  • Awesome! I just used this same technique to spin up [multiple delayed_job processes on a single worker dyno](http://stackoverflow.com/a/21568770/165673) – Yarin Feb 05 '14 at 04:41
  • Does Heroku monitor the processes started by the `Procfile.workers` file? That's the core problem with not using resque's `COUNT=2` setting. – ajsharma Sep 06 '16 at 19:20
  • I suppose if one of the workers dies the foreman running the "Procfile.workers" should kill all of its children and terminate itself - and this should then become visible to heroku. I was running the above setup for a while without problems, without problems, but at some point decided to just go with heroku's defaults. – radiospiel Sep 08 '16 at 13:09
  • If I follow the steps, heroku cashes the processes. Further foreman gem mentions that, "Ruby users should take care not to install foreman in their project's Gemfile." – Bot Aug 02 '17 at 11:20
  • Hi @Bot: ok, there you go. First, if heroku "crashes" the processes than your application is quite likely over the RAM usage limit. Since this approach starts more than the one process that heroku usually starts from your Procfile this would multiply the RAM requirements, therefore you might run into to 512 MB limit exposed by the small heroku instances. – radiospiel Aug 04 '17 at 07:41
  • Second: "Ruby users should take care not to install foreman in their project's Gemfile." Well, Gemfile is the only way to install foreman on a heroku machine (well, you could use a buildpack, that's a different matter.) – radiospiel Aug 04 '17 at 07:42
1

Based on this article, it sounds like it's possible, but the biggest gotcha is that if one of the child processes dies, Heroku won't be able to restart it.

CDub
  • 13,146
  • 4
  • 51
  • 68
  • Thanks for your effort. I did find a way which is easier to build and probably more stable also - see above – radiospiel Nov 04 '13 at 09:02