20

i created a celeryd file in /etc/defaults/ from the code here:

https://github.com/celery/celery/blob/3.0/extra/generic-init.d/celeryd

Now when I want to run celeryd as a daemon and do this: sudo /etc/init.d/celerdy it says command not found. Where am I going wrong?

octosquidopus
  • 3,517
  • 8
  • 35
  • 53
Hick
  • 35,524
  • 46
  • 151
  • 243

5 Answers5

22

I am not sure what you are doing here but these are the steps to run celery as a daemon.

  1. The file that you have referred in the link https://github.com/celery/celery/blob/3.0/extra/generic-init.d/celeryd needs to be copied in your /etc/init.d folder with the name celeryd
  2. Then you need to create a configuration file in the folder /etc/default with the name celeryd that is used by the above script. This configuration file basically defines certain variables and paths that are used by the above script. Here's an example configuration.
  3. This link Generic init scripts explains the process and can be used for reference
ppython
  • 485
  • 6
  • 19
Rohan
  • 1,031
  • 7
  • 10
  • by @Kishor-Pawar : I wish to add something to your answer. Once you create both the files, you want to start celeryd service.(service celeryd start) It may throw error like invalid user 'celery'. which is mentioned at conf file (/etc/default/celeryd) with CELERY_USER. You may want to change specified user to your current user or need to create new user. You also need to provide required privileges. – bummi Apr 17 '14 at 08:51
  • 2
    If you read the instructions on the celery page, it is no wonder people get confused as the implication is there is one file and it goes in /etc/default. For people with serverside linux experience you can work it out, but any novice is going to be lost. This posted answer should go in the instructions – MagicLAMP Oct 03 '16 at 03:59
9

I found this link extremely useful: How to write an Ubuntu Upstart job for Celery (django-celery) in a virtualenv

tweaking it a bit.. I have a celery worker running using this script:

(using ubuntu upstart)

named iamcelery.conf and placed it in /etc/init (note: not init.d)

# iamcelery -runs the celery worker as my virtual env user
#
#
# This task is run on startup to start the celery worker as my vritual env user

description "runs the celery worker"
author "michel van Leeuwen <michel@iamit.nl>"

start on runlevel [2345]
stop on runlevel [!2345]

# retry if ended unexpectedly
respawn
# limit the retries to max 15 times with timeouts of 5 seconds
respawn limit 15 5

# Time to wait between sending TERM and KILL signals
kill timeout 20

task
script
  exec su -s /bin/sh -c 'exec "$0" "$@"' <place here your unprovilegd username> -- srv/<here the path of your django project>/bin/django celeryd -BE -l info
end script

now you can start this scipt (it starts on server startup as well):

sudo start iamcelery

or stop:

sudo stop iamcelery

or check its status:

sudo status iamcelery

I am not quit sure this is the neatest way.... however... after a long trial and errors trying to get the initd scripts to work.... ( without succes) ... this finally works.

Edit 8 june 2013 My script given here seemed to runs as a root in the end. Now I changed this:

script
  su <place here your unprovilegd username>
  cd /srv/<here the path of your django project>/
  exec bin/django celeryd -BE -l info
end script

into:

script
  exec su -s /bin/sh -c 'exec "$0" "$@"' <place here your unprovilegd username> -- srv/<here the path of your django project>/bin/django celeryd -BE -l info
end script

and this works, with all the credits to the answer to this question: How to write an Ubuntu Upstart job for Celery (django-celery) in a virtualenv

Edit 5 sept 2013

There is one small thing left: I have to do ctrl-c after the start command in the console (and do a status check after this one): In case somebody knows this: leave in the command, and I can update this answer...

Community
  • 1
  • 1
michel.iamit
  • 5,788
  • 9
  • 55
  • 74
9

I generally use supervisor (plus django-supervisor) for this purpose. That way, you don't need to figure out how to daemonize each process in your application (of which you have at least a webserver hosting django, plus celery, plus realistically whatever other middleware you use to support both of those). Supervisor knows how to run itself as a daemon, and all your other processes run as children of supervisor.

Marcin
  • 48,559
  • 18
  • 128
  • 201
  • so you install supervisor, and then what? – frankster Nov 18 '16 at 17:18
  • 3
    @frankster Then you use it – Marcin Nov 18 '16 at 17:20
  • liunx is totally a mess,I got lots of choice as init script and systemd and ubuntu upstart all works.but why we need to learn all that >>> instead we use supervisor works well on all liunx distro and only learn for the first time.. –  Mar 30 '17 at 02:13
4

As Marcin has explained in his answer that supervisor is usually what people end up using but if you are looking for something which could work with python3 and can't wait for supervisor's version 4 which I think will have the support for python3 then you can go with circus. After installing it, you just need to have a circus.ini file which will have all the processes which you want to daemonize and then run that sample circus.ini may look like:

[watcher:celery]
cmd = full_path/python3.4 full_path/manage.py celeryd -B -l info

[watcher:celerycamera]
cmd = full_path/python3.4 full_path/manage.py celery events --camera=djcelery.snapshot.Camera

[watcher:dceleryflower]
cmd = full_path/python3.4 full_path/manage.py celery flower -A your_app_name --basic_auth=username:password --port=5555

if you want some more details I have a post related to the same here. Hope it saves someone some time. Thanks

Aameer
  • 1,366
  • 1
  • 11
  • 30
  • You can run any kind of processes through supervisor, whatever version of python runs supervisor. Of course, then you would need two virtualenvs which may not be totally desirable. – Marcin Nov 18 '16 at 17:21
1

Note: in ubuntu 16.04 my anser with the .conf file is not working anymore.

I created a .service file and put this in /etc/systemd/system/

i can use

sudo service myservice status

sudo service myservice start

sudo service myservice stop

as commands

e.g. this file:

myservice.service:

[Unit] 
Description=My celery worker 

[Service]
WorkingDirectory=/srv/my-project-path
User=buildout
Group=buildout
Restart=on-failure
RestartSec=20 5
ExecStart=/srv/my-project/bin/django celeryd -BE

[Install]
WantedBy=multi-user.target
Alias=myservice.service

note i use buildout, so in setad of bin/django most users need to use the path to python and use mange.py in stead.

base upon: http://minecraft.gamepedia.com/Tutorials/Ubuntu_startup_script (see the with systemd section)

michel.iamit
  • 5,788
  • 9
  • 55
  • 74