5

All, I'm attempting to 'force' RQ workers to perform concurrently using supervisord. My setup supervisord setup seems to work fine, as rq-dashboard is showing 3 workers, 3 PID's and 3 queue (one for each worker/PID). Supervisord setup is as follows (showing only worker 1 setup, 2 more workers are defined below this one):

[program:rqworker1]
command = rqworker 1 
process_name = rqworker1-%(process_num)s
numprocs = 1 
user = username 
autostart = True
stdout_logfile=/tmp/rqworker1.log
stdout_logfile_maxbytes=50MB

RQ workers running under supervisord

The issue is when I send 3 jobs concurrently, the total time to run is x3 that of a single task (namely, total time is linear with number of tasks, this scales to x4,x5, etc..). It seems no concurrency is available. I also implemented a primitive load-balancing by sending new jobs to the queue with minimum started+queued jobs, that works fine (jobs are observed to be spread evenly among queues).

Why would this setup not allow concurrency?

Any considerations regarding the setup i'm missing?

Note that rq-gevent-worker package (which worked great earlier w.r.t. concurrency/RQ) is no longer available as I migrated to PY3 and gevent itself is not yet supported on PY3. But this gives my a clue that concurrency is possible.

GG_Python
  • 3,436
  • 5
  • 34
  • 46
  • Maybe you're feeding all the jobs to the same worker/queue? – John Mee Apr 30 '15 at 02:19
  • nah, i've tested this; first, each worker monitors a single queue, what i do is spread the jobs to the three queues evenly (again, tested to work), from there each worker is supposed to only handle its own job(s). But this is not done concurrently, as time to finish is linear with number of jobs. If concurrency worked, one would get a total time = #Jobs/#Workers, assuming jobs are evenly distributed among queues/workers. – GG_Python Apr 30 '15 at 03:03
  • 2
    I'm using `supervisord` to run multiple rqworkers in parallel. (This isn't a "force", by the way, it's an [intended pattern](http://python-rq.org/patterns/supervisor/) in `python-rq`.) On the other hand, writing your own load balancing algorithm is an anti-pattern: that's exactly what `python-rq` does for you. If you want to split up work between 3 workers, then they should all listen to the same queue. You can remove 2 of your supervisor config blocks and change `numprocs` to `3`. Otherwise, there's not enough information to know what's wrong: how many CPUs? is the task IO bound or CPU bound? – Mark E. Haase Aug 30 '15 at 14:59
  • @mehaase, got around to checking this, got rid of 'load balancing', and you're right, this works. Your comment above should be the accepted answer. – GG_Python Jan 05 '16 at 17:33

1 Answers1

4

Modifying my comment from above into an answer...

Using supervisord to run multiple rqworker processes in parallel is an intended pattern in python-rq, so don't be concerned that you're "forcing" it. You actually have the right idea.

On the other hand, writing your own load balancing algorithm is an anti-pattern: that's exactly what python-rq does for you.

If you want to split up work between three workers, then they should all listen to the same queue. Try removing two of your supervisor config blocks, and in the one block that remains, change numprocs to 3. If you submit three jobs rapidly to that queue, you should see three workers executing concurrently.

Mark E. Haase
  • 25,965
  • 11
  • 66
  • 72