delayed_job with multiple workers and upstart

Question

I want to switch my single delayed_job process to multiple workers. I currently have an upstart job that runs rake and uses respawn method with no 'expect fork' since rake does not fork. Now to switch to a multiple worker method I need to 'expect' in my upstart configuration file. Any suggestions.

Answer 1

Out of the box, it appears that upstart expect does not support the behavior outlined in https://github.com/collectiveidea/delayed_job#running-jobs , as there are multiple workers that each fork twice to daemonize.

As outlined in this question about upstart: Can upstart expect/respawn be used on processes that fork more than twice? , you can use a bit of scripting to shepherd the processes yourself in the different hooks.

Another option would be to use upstart job instances (http://upstart.ubuntu.com/cookbook/#instance) to start multiple jobs that do not fork.

Answer 2

I'm not very clear with what you were asking. But if you want multiple delayed jobs to run in background, when you start the delayed job using the command something like rake Jobs:Work, you can specify the number of consumer threads you want to spawn. Hope it helps you.