0

I'm running celery 3.1.25

my supervisor .conf file is this:

[group:proj]
programs=celerycam,celerybeat,celeryd-urgent,celeryd-default,celeryd-temp
priority=999

[program:gunicorn]
process_name=%(program_name)s
autorestart=true
command=/home/proj/.virutalenvs/proj_env/bin/gunicorn_django -c /home/proj/www/proj/gunicorn_conf.py
directory = /home/proj/www/proj
user = proj
stdout_logfile = /var/log/supervisor/proj/%(program_name)s.log
stderr_logfile = /var/log/supervisor/proj/error-%(program_name)s.log
stdout_logfile_maxbytes=25MB
stdout_logfile_backups=5
stderr_logfile_maxbytes=25MB
stderr_logfile_backups=5

[program:celerycam]
process_name=%(program_name)s
autorestart=true
command=/home/proj/.virtualenvs/proj/bin/python /home/proj/www/proj/manage.py celerycam
stdout_logfile = /var/log/supervisor/proj/%(program_name)s.log
stderr_logfile = /var/log/supervisor/proj/error-%(program_name)s.log
user = root
password = proj
stdout_logfile_maxbytes=25MB
stdout_logfile_backups=5
stderr_logfile_maxbytes=25MB
stderr_logfile_backups=5

[program:celeryd-temp]
process_name=%(program_name)s
autorestart=true
exitcodes=0,2
directory=/home/proj/www/proj
command=/home/proj/.virtualenvs/proj_env/bin/python /home/proj/www/proj/manage.py celery worker -E --    loglevel=DEBUG -Q urgent -n urgent --concurrency=8 --maxtasksperchild=1
stdout_logfile = /var/log/supervisor/proj/%(program_name)s.log
stderr_logfile = /var/log/supervisor/proj/%(program_name)s.log
stdout_logfile_maxbytes=25MB
stdout_logfile_backups=8
stderr_logfile_maxbytes=25MB
stderr_logfile_backups=8

[program:celeryd-default]
process_name=%(program_name)s
autorestart=true
directory=/home/proj/www/proj
command=/home/proj/.virtualenvs/proj_env/bin/python /home/proj/www/proj/manage.py celery worker -E --        loglevel=DEBUG -Q default -n default --concurrency=8 --maxtasksperchild=1
stdout_logfile = /var/log/supervisor/proj/%(program_name)s.log
stderr_logfile = /var/log/supervisor/proj/%(program_name)s.log
stdout_logfile_maxbytes=25MB
stdout_logfile_backups=5
stderr_logfile_maxbytes=25MB
stderr_logfile_backups=5

I the supervisor and everything looks fine, but when I check the status

`sudo supervisorctl status`

I get the following:

proj:celerybeat             RUNNING   pid 13030, uptime 0:13:27
proj:celerycam              RUNNING   pid 13015, uptime 0:13:28
proj:celeryd-default        RUNNING   pid 18845, uptime 0:00:02
proj:celeryd-temp           STARTING

If I check status again, I get the following

proj:celerybeat             RUNNING   pid 13030, uptime 0:15:13
proj:celerycam              RUNNING   pid 13015, uptime 0:15:14
proj:celeryd-default        STARTING
proj:celeryd-temp           RUNNING   pid 19512, uptime 0:00:01

This is what sudo supervisorctl tail proj:celeryd-default prints:

r removal in
version 4.0. Please use "group" instead (see the Canvas section in the userguide)

""")
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

/home/csanalytics/.virtualenvs/proj_env/local/lib/python2.7/site-    packages/celery/task/sets.py:23: CDeprecationWarning:
    celery.task.sets and TaskSet is deprecated and scheduled for removal in
version 4.0. Please use "group" instead (see the Canvas section in     the userguide)

  """)

I can run the commands on the supervisor file from my terminal without a problem, but for some reason they are crashing on supervisor. Any idea?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
user3007270
  • 406
  • 4
  • 15

1 Answers1

0

This is running workers as a superuser and this doesn't work well with the newer versions of Celery.

More specifically, workers from Celery 3.1+ don't work well with pickle serialization.

You will need to disable pickle serialization in celery configs

app.conf.update(
 CELERY_ACCEPT_CONTENT = ['json'],
 CELERY_TASK_SERIALIZER = 'json',
 CELERY_RESULT_SERIALIZER = 'json',
)

and run it.

user3007270
  • 406
  • 4
  • 15