3

I seem to have tried every solution on here but none seem to be working, I'm not sure what I'm missing. Im trying to run celery as a daemon through my docker container.

root@bae5de770400:/itapp/itapp# /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
root@bae5de770400:/itapp/itapp# /etc/init.d/celerybeat status
celery init v10.1.
Using configuration: /etc/default/celeryd
celerybeat is down: no pid file found
root@bae5de770400:/itapp/itapp#

I've seen lots of posts to do with perms and I've tried them all to no avail.

this is my docker file which creates all the perms and folders

FROM python:latest
ENV PYTHONUNBUFFERED 1

# add source for snmp
RUN sed -i "s#jessie main#jessie main contrib non-free#g" /etc/apt/sources.list
# install dependancies
RUN apt-get update -y \
    && apt-get install -y apt-utils python-software-properties libsasl2-dev python3-dev libldap2-dev libssl-dev libsnmp-dev snmp-mibs-downloader git vim

# copy and install requirements
RUN mkdir /config  
ADD /config/requirements.txt /config/  
RUN pip install -r /config/requirements.txt  
# create folders
RUN mkdir /itapp;
RUN mkdir /static;
# create celery user
RUN useradd -N -M --system -s /bin/false celery
RUN echo celery:"*****" | /usr/sbin/chpasswd
# celery perms
RUN groupadd grp_celery
RUN usermod -a -G grp_celery celery
RUN mkdir /var/run/celery/
RUN mkdir /var/log/celery/
RUN chown root:root /var/run/celery/
RUN chown root:root /var/log/celery/
# copy celery daemon files
ADD /config/celery/init_celeryd /etc/init.d/celeryd
RUN chmod +x /etc/init.d/celeryd
ADD /config/celery/celerybeat /etc/init.d/celerybeat
RUN chmod +x /etc/init.d/celerybeat
RUN chmod 755 /etc/init.d/celeryd
RUN chown root:root  /etc/init.d/celeryd
RUN chmod 755 /etc/init.d/celerybeat
RUN chown root:root /etc/init.d/celerybeat
# copy celery config
ADD /config/celery/default_celeryd /etc/default/celeryd
# RUN /etc/init.d/celeryd start
# set workign DIR for copying code
WORKDIR /itapp

if I start it manually it works

celery -A itapp worker -l info
/usr/local/lib/python3.6/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the -u option.
...

[2017-09-25 17:29:51,707: INFO/MainProcess] Connected to amqp://it-app:**@rabbitmq:5672/it-app-vhost
[2017-09-25 17:29:51,730: INFO/MainProcess] mingle: searching for neighbors
[2017-09-25 17:29:52,764: INFO/MainProcess] mingle: all alone

the init.d files are copied from the celery repo and this is the contents of my default file if it helps

# Names of nodes to start
#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="itapp"
# or fully qualified:

# Where to chdir at start.
CELERYD_CHDIR="/itapp/itapp/"

# Extra command-line arguments to the worker
CELERYD_OPTS="flower --time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

# Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

the only thing in this file which may be wrong I think is the CELERY_BIN value, I'm not sure what to set that too in a docker container

Thanks

AlexW
  • 2,843
  • 12
  • 74
  • 156

1 Answers1

4

So you had few issues in your Dockerfile

  • Celery process shell was set to /bin/false which didn't allow any process to be started.
  • You needed to give permission on /var/run/celery and /var/log/celery to the celery user
  • /etc/default/celeryd should be 640 permission
  • Also too many layers in your Dockerfile

So I updated the Dockerfile to below

FROM python:latest
ENV PYTHONUNBUFFERED 1

# add source for snmp
RUN sed -i "s#jessie main#jessie main contrib non-free#g" /etc/apt/sources.list
# install dependancies
RUN apt-get update -y \
    && apt-get install -y apt-utils python-software-properties libsasl2-dev python3-dev libldap2-dev libssl-dev libsnmp-dev git vim

# copy and install requirements
RUN mkdir /config
ADD /config/requirements.txt /config/
RUN pip install -r /config/requirements.txt
# create folders
RUN mkdir /itapp && mkdir /static;
# create celery user
RUN useradd -N -M --system -s /bin/bash celery && echo celery:"B1llyB0n3s" | /usr/sbin/chpasswd
# celery perms
RUN groupadd grp_celery && usermod -a -G grp_celery celery && mkdir -p /var/run/celery/ /var/log/celery/
RUN chown -R celery:grp_celery /var/run/celery/ /var/log/celery/
# copy celery daemon files
ADD /config/celery/init_celeryd /etc/init.d/celeryd
RUN chmod +x /etc/init.d/celeryd
ADD /config/celery/celerybeat /etc/init.d/celerybeat
RUN chmod 750 /etc/init.d/celeryd /etc/init.d/celerybeat
RUN chown root:root  /etc/init.d/celeryd /etc/init.d/celerybeat
# copy celery config
ADD /config/celery/default_celeryd /etc/default/celeryd
RUN chmod 640 /etc/default/celeryd
# set workign DIR for copying code
ADD /itapp/ /itapp/itapp
WORKDIR /itapp

And then got into the web service container and all worked fine

root@ab658c5d0c67:/itapp/itapp# /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
root@ab658c5d0c67:/itapp/itapp# /etc/init.d/celeryd start
celery init v10.1.
Using config script: /etc/default/celeryd
celery multi v4.1.0 (latentcall)
> Starting nodes...
    > worker1@ab658c5d0c67: OK
    > flower@ab658c5d0c67: OK
root@ab658c5d0c67:/itapp/itapp# /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
root@ab658c5d0c67:/itapp/itapp# /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd (node worker1) (pid 66) is up...
root@ab658c5d0c67:/itapp/itapp#
Tarun Lalwani
  • 142,312
  • 9
  • 204
  • 265
  • Thanks for your help! celeryd now starts but celerybeat still has the same issue, whats missing to get them both up? – AlexW Sep 26 '17 at 15:47
  • any ideas on this one? – AlexW Sep 27 '17 at 08:18
  • Run this command in your container `/usr/local/bin/celery beat --app=itapp -f /var/log/celery/beat.log -l INFO --workdir=/itapp/itapp/ --pidfile=/var/run/celery/beat.pid` and you will find that you have missing packages. So fix that in your requirements.txt first – Tarun Lalwani Sep 27 '17 at 08:33
  • PermissionError: [Errno 13] Permission denied: '/var/log/celery/beat.log' I thought settings were already set in the docker fie though? – AlexW Sep 27 '17 at 12:02
  • root@70f24b238184:/itapp/itapp# namei -l /var/log/celery/beat.log f: /var/log/celery/beat.log drwxr-xr-x root root / drwxr-xr-x root root var drwxr-xr-x root root log drwxr-xr-x celery grp_celery celery -rw-r--r-- root root beat.log root@70f24b238184:/itapp/itapp# – AlexW Sep 27 '17 at 12:06
  • Try `su celery -c "/usr/local/bin/celery beat --app=itapp -f /var/log/celery/beat.log -l INFO --workdir=/itapp/itapp/ --pidfile=/var/run/celery/beat.pid"` – Tarun Lalwani Sep 27 '17 at 12:13
  • that started celery beat without issues – AlexW Sep 27 '17 at 12:17
  • then I hope your issue is fixed – Tarun Lalwani Sep 27 '17 at 12:20
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/155419/discussion-between-tarun-lalwani-and-alexw). – Tarun Lalwani Sep 27 '17 at 12:22
  • @tarun-lalwani @AlexW If I restart my docker container and do `sudo service celeryd status` , I get the below output. `celery init v10.1. Using config script: /etc/default/celeryd celeryd (node process_log__worker) (pid 177) is down, but pidfile exists! ` Any help?? I expect the celery auto-restart upon my docker container reboot. Which is not happening for me. I have followed the same steps, everything is working except the reboot part. Any help will be appreciated – CoooolllYuvee Mar 20 '18 at 13:35
  • im not sure from that, I do recall seeing that error when I was trying to get it running but cannot remember how I fixed it. try post a question on here. someone with more experience will be able to help you out – AlexW Mar 20 '18 at 14:35