1

So, I've been given a code base which uses daemons, daemons-rails and delayed jobs to trigger a number of *_ctl files in /lib/daemons/ but here's the problem:

If two people do an action which starts the daemons doing some heavy lifting then whichever one clicks second will have to wait for the first to complete. Not good. We need to start multiple daemons on each queue.

Ideally what I want to do is read a config file like this:

default:
  queues: default ordering 
  num_workers: 10
hours:
  queues: slow_admin_tasks
  num_workers: 2
minutes:
  queues: minute
  num_workers: 2

This would mean that 10 daemon processes are started to listen to the default and ordering queues, 2 for slow_admin tasks etc.

How would I go about defining multiple daemons like this, it looks like it might be in one of these places:

  • /lib/daemons/*_ctl
  • /lib/daemons/*.rb
  • /lib/daemons/daemons

I thought it might be a change to the daemons-rails rake tasks, but they just hit the daemons file.

Has anyone looked in to scaling daemons-rails in this way? Where can I get more information?

AJFaraday
  • 2,411
  • 1
  • 16
  • 39

1 Answers1

0
  1. I suggest you to try Foreman.

Take a look at this discution.

Foreman can help manage multiple processes that your Rails app depends upon when running in development. You can find a tutorial regarding Foreman on RailsCasts. There's a video tutorial + some source code examples.

  1. I also suggest you to take a look at Sidekiq.

Sidekiq allows you to move jobs into the background for asynchronous processing. It uses threads instead of forks so it is much more efficient with memory compared to Resque. You can find a tutorial here.

  1. I also suggest you to take a look at Resque

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. You can find a tutorial here.

Community
  • 1
  • 1
Pascut
  • 3,291
  • 6
  • 36
  • 64