20

I am using delayed_job and moved to a new beefier server. So now I would like to run parallel jobs, as now I have the POWER!, but am confused on whether delayed_job can run multiple parallel queues?

This question suggested that there are named queues, but do these all run off the one table and are thus sequential?

At the bottom @Jesse Wolgamott suggests that you can create a table for each queue that will then run in parrallel.

Has anyone done this and can they point me to how it is done?

Community
  • 1
  • 1
slotishtype
  • 2,715
  • 7
  • 32
  • 47
  • No, multiple workers works in parallel not sequential. Even if the queues are all in one table, workers will use sql query to select jobs queue it is assigned to, order by `run_at`, `priority`. Therefore multiple workers can work on different parts in that table in parallel. – lulalala Aug 16 '12 at 02:57
  • I forgot about the answer, so I added what I use and accepted it. I also upvoted the other... – slotishtype Aug 16 '12 at 13:31

3 Answers3

32

It is possible and I am doing it all the time. In our case we need multiple jobs for processing three different kinds of jobs say queue_a, queue_b, and queue_c. The system will make entries in the delayed_job table and the queue named appropriately.

Start multiple delayed jobs like

RAILS_ENV=production script/delayed_job -i first --queue=queue_a start
RAILS_ENV=production script/delayed_job -i second --queue=queue_a start
RAILS_ENV=production script/delayed_job -i third --queue=queue_b  start
RAILS_ENV=production script/delayed_job -i fourth --queue=queue_c start

Each command will create a delayed_job, so there will be now 4 parallel jobs, two of them serving queue_a and one each for queue_b and queue_c. The key thing here is the identifier that is passed through the -i option which specifies the instance name, and we can start and stop jobs as required.

Another option is to use worker pools.

RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start

That command will start 1 worker for the tracking queue, 2 workers for the mailers and tasks queues, and 2 workers for any jobs.

3coins
  • 1,322
  • 3
  • 11
  • 30
Ggmon
  • 638
  • 1
  • 7
  • 12
  • Do you happen to know, if the last one (2 workers for any jobs) will also take jobs from the other queues (tracking, mailers, tasks) or only jobs where no queue was specified? – Machisuji Mar 24 '15 at 09:48
  • I think there is a correction needed here for specifying multiple instances. The syntax should be like this RAILS_ENV=production script/delayed_job --queue=queue_a -i one1 start, notice the space between -i and instance name. – 3coins Sep 21 '15 at 20:54
  • Thank you, this one worked very well for me... thanks! I am yet to check out the -n option... – Gnana Jun 02 '16 at 13:29
  • It should be noted that the -i parameter only work with --queue, and does not work with --pool – Emiliano Della Casa Apr 12 '18 at 14:51
  • I have already added delay in multiple places ("around 80"), I want to add new delay with 2 specific workers with using `queue` How can i achieve without adding `queue` in past other delay's. – Vishwas Nahar Apr 26 '19 at 06:34
16

With bundler in production:

RAILS_ENV=production bundle exec script/delayed_job -n 4 start

or without bundler

ruby script/delayed_job -n 4 start

Purag
  • 16,941
  • 4
  • 54
  • 75
slotishtype
  • 2,715
  • 7
  • 32
  • 47
  • 1
    I have not tried the -i option but yet the -n option does not seem to work when you are doing specifying a queue. You will get multiple workers running but only one worker will apply to an actual thread. – Dex May 10 '14 at 22:54
  • "-n 4" - This is big mistake! Use -n4, without space! Waisted several hours on that. – Andrew Rukin May 26 '15 at 12:54
  • @AndrewRukin the documentation explicitly mentions -n 2 (with space...) see here: https://github.com/collectiveidea/delayed_job#running-jobs – rept Jan 27 '16 at 00:15
  • Is this possible to use `queue` and delayed job `worker` picks only `queue` one not the other free `workers` ? – Vishwas Nahar Apr 26 '19 at 06:38
4

If you're using rake tasks as your job launch mechanism, the QUEUE environment variable can be used to launch different worker jobs separated by queue.

Example:

QUEUE=email rake jobs:work
QUEUE=build_data rake jobs:work

The QUEUES variable lets one dequeue from multiple DJ queues for a particular worker.

QUEUES=build_data,email rake jobs:work

Particularly useful if you're using a hosted/cloud environment (e.g. Heroku) that gives you limited access to launching scripts/jobs directly.

changingrainbows
  • 2,551
  • 1
  • 28
  • 35
  • Not working, if i set my multiple parallel job queues as follows: `QUEUES=build_data,email rake jobs:work` and then `QUEUE=another_data rake jobs:work` – Peter Prabu Nov 27 '14 at 08:55