Celery jobs not running

Question

When I run m script like this on ubuntu as root, it works:

./my_celery.sh

But when it runs with Upstart like this it does not execute the task:

Upstart:

description "Django instance"
start on runlevel [2345]
stop on runlevel [06]
respawn
respawn limit 10 5
exec /var/www/webapps/myapp/my_celery.sh

Start it:

service my_celery start

Log:

[2015-07-20 11:56:46,563: INFO/MainProcess] Scheduler: Sending due task The Parmalat Daily Report (minamobime_coza.tasks.parmalat_report)

The tasks are never executed

Answer 1

I had trouble getting upstart to run celery workers - in fact upstart never quite managed to live up to its promise.

I would recommend using supervisord instead of upstart to manage the celery workers - example config file:

/etc/supervisor.d/celery.conf

[program:celery]
command=celery worker --app=path.to.my.tasks    
user=celery
autostart=true
stdout_logfile=/var/log/supervisor/celery.log
stdout_logfile_maxbytes=250MB
stdout_logfile_backups=10
redirect_stderr=true