2

Can i run delayed_job or similar schedule frameworks inside of the web server eg. thin or unicorn? If yes how do i start it? (code example would be very cool!)

The reason is that i want to save money during my application is just in a build-up phase and it is hosted on heroku.

SeriousM
  • 3,374
  • 28
  • 33

3 Answers3

3

Officially

No, there is no supported way to run delayed_jobs asynchronously within the web framework. From the documentation on running jobs, it looks like the only supported way to run a job is to run a rake task or the delayed job script. Also, it seems conceptually wrong to bend a Rack server, which was designed to handle incoming client requests, to support pulling tasks off of some queue somewhere.

The Kludge

That said, I understand that saving money sometimes trumps being conceptually perfect. Take a look at these rake tasks. My kludge is to create a special endpoint in your Rails server that you hit periodically from some remote location. Inside this endpoint, instantiate a Delayed::Worker and call .start on it with the exit_on_complete option. This way, you won't need a new dyno or command.

Be warned, it's kind of a kludgy solution and it will tie up one of your rails processes until all delayed jobs are complete. That means unless you have other rails processes, all incoming requests will block until this queue request is finished. Unicorn provides facilities to spawn worker processes. Whether or not this solution will work will also depend on your jobs and how long they take to run and your application's delay tolerances.

Edit

With the spawn gem, you can wrap your instantiation of the Delayed::Worker with a spawn block, which will cause your jobs to be run in a separate process. This means your rails process will be available to serve web requests immediately instead of blocking while delayed jobs are run. However, the spawn gem has some dependencies on ActiveRecord and I do not know what DB/ORM you are using.

Here is some example code, because it's becoming a bit hazy:

class JobsController < ApplicationController
  def run
    spawn do
      @options = {} # youll have to get these from that rake file
      Delayed::Worker.new(@options.merge(exit_on_complete: true)).start
    end
  end
end
accounted4
  • 1,685
  • 1
  • 11
  • 13
  • Well explained, thank you. What if I create an Daemon of my code, would that block my thread/worker? https://github.com/ghazel/daemons – SeriousM Feb 18 '13 at 17:39
  • I think that the Daemon plugin is a way to convert a script with a loop in it into something that can run in the background and be controlled from the command line (that is, you can call `stop` to stop your background process). I would suggest looking into [spawn](https://github.com/tra/spawn) instead. – accounted4 Feb 18 '13 at 17:44
  • Interesting! With that I could spawn a new thread, run my processes (maybe also in a new thread) and start over? sounds good! Btw: I wouldn't do that for too long. I just need to know if the project makes sense before I spend some money for it. – SeriousM Feb 18 '13 at 17:47
  • I'm using mongodb. Do i really need the exit_on_complete when the thing should work for ever? – SeriousM Feb 18 '13 at 17:55
  • If you are thinking about spawning a never-ending process, you can ditch `exit_on_complete` and do your spawn block in an initializer. Then you won't need the endpoint or anything. – accounted4 Feb 18 '13 at 19:21
  • yes, that was my intention. would this process die when the main process exits? sorry when i ask no-brainer questions but im not so experienced with ruby's process/thread model. – SeriousM Feb 19 '13 at 09:05
  • 1
    I recommend asking a separate question about the spawn gem and how it works. I know that it actually uses processes by default instead of threads. But I don't know how it will interact with your environment. I also recommend reading the spawn docs. – accounted4 Feb 19 '13 at 19:42
1

Here's a link to a similar question:

Is it feasible to run multiple processeses on a Heroku dyno?

Bear in mind, as the post says, if you're only using one web dyno, it will be shut down if there's no traffic going to it.

In a similar vein, you might look into:

http://blog.codeship.io/2012/05/06/Unicorn-on-Heroku.html

To save on the need for multiple web dynos whilst you're building your app (although it's still subject to the above shutdown issue).

I would suggest you might look at running on a VPS directly, rather than Heroku (check out the railscast):

http://railscasts.com/episodes/337-capistrano-recipes

Once set up, it's pretty easy to deploy to. Heroku cuts out the devops part for you.

Community
  • 1
  • 1
Tom
  • 68
  • 8
  • Thanks for that but I have a NewRelic profiler running that keeps my dyno online. Btw your answer does not really answer my question if i can run delayed_job in the web process (bonus: and how). Could you help me with that please? – SeriousM Feb 18 '13 at 16:57
  • The first link answers if you can (Yes!), but I wouldn't want to (the dynos are really quite restricted in memory and cpu). I'd save money with multiple Unicorn workers (2-4 concurrent requests depending on the size of your app), giving you the money to get a worker dyno up. – Tom Feb 18 '13 at 17:12
  • I'm using unicorn already but want to have a Worker for free too :). How can I start a process/thread inside unicorn? – SeriousM Feb 18 '13 at 17:15
  • Haven't found anything specifically to do what you're looking for, but there is https://github.com/meskyanichi/hirefire (A gem that automatically creates a worker when there is a job in queue, and removes it when the jobs are complete) - should massively cut down on the cost of a worker dyno. – Tom Feb 18 '13 at 17:26
  • hirefire sounds like a nice solution but you have to pay ~10$ for the service (the gem is only for the connection to the service.) – SeriousM Feb 18 '13 at 17:38
0

You can run it inside a separate worker of Unicorn, so it shares memory with the master process and get restarted together with the app.

See https://gist.github.com/brauliobo/11298486

brauliobo
  • 5,843
  • 4
  • 29
  • 34