How can I create a large number of rq worker processes in a VPS easily?
Right now I'm manually opening a terminal and running python3 worker.py in it, and then repeating this until I get a satisfying number of worker instances running. I know this is not a scalable solution, so how can I do it automatically and easily. It'd be nice if there were some tool which facilitates this process.
I ended up using Supervisord. It seemed like a pretty good solution.
The relevant supervisord.conf file looks like this.
[supervisord]
[program:worker]
command=python worker.py
process_name=%(program_name)s-%(process_num)s
numprocs=20
directory=.
stopsignal=TERM
autostart=true
autorestart=true
After putting this file in the same directory as of the RQ worker script worker.py, just running the following command will spawn up 20 instances of the RQ worker. The number of workers can be specified by the numprocs option in the supervisord.conf file.
$ supervisord -n