how to detect failure and auto restart celery worker

Question

I use Celery and Celerybeat in my django powered website. the server OS is Ubuntu 16.04. by using celerybeat, a job is done by a celery worker every 10 minutes. sometimes the worker shuts down without any useful log messages or errors. So, I want to find a way in order to detect status (On/Off) of celery worker (not Beat), and if it's stopped, restart it automatically. how can I do that? thanks

Answer 1

In production, you should run Celery, Beat, your APP server etc. as daemons [1] using Supervisor/Upstart/Systemd/.../.... There is a section about this in the Celery documentation. [2]

My favorite tool is Supervisord [3]. Here is Supervisord example configuration for Celery: https://github.com/celery/celery/tree/master/extra/supervisord and here https://github.com/illagrenan/ubuntu-supervisor-configuration is a tutorial about installing Supervisord on Ubuntu.

(...) Supervisord starts processes as its subprocesses, and can be configured to automatically restart them on a crash. (...)

Source: http://supervisord.org/introduction.html#introduction

• [1] https://en.wikipedia.org/wiki/Daemon_(computing)
• [2] http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
• [3] http://supervisord.org/