HEALTHCHECK of a Docker container running Celery tasks?

Question

I know one of the ways to check health for Docker container is using the commmand

HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1

But in case of workers there is no such URL to hit , How to check the container's health in that case ?

Answer 1

The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.

Assuming your app is named tasks.add, you may ping all your workers:

/app $ celery inspect ping -A tasks.add
-> celery@aa7c21dd0e96: OK
        pong
-> celery@57615db15d80: OK
        pong

With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.

To ping a single node, you would have to run:

celery inspect ping -A tasks.add -d celery@$HOSTNAME

Here, d stands for destination.

The line to add to your Dockerfile:

HEALTHCHECK CMD celery inspect ping -A tasks.add -d celery@$HOSTNAME

---

Sample outputs:

/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69

Unhealthy if the node does not exist or does not reply

/app $ celery inspect ping -A tasks.add -d celery@$HOSTNAME
-> celery@d39b3d31cc13: OK
        pong
/app $ echo $?
0

Healthy when the node replies pong.

/app $ celery inspect ping -d celery@$HOSTNAME
Traceback (most recent call last):
...
    raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1

Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails This might not suit your needs, the broker is unhealthy, not the worker.

Answer 2

The below example snippet, derived from that posted by @PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.

worker:
    build:
        context: .
        dockerfile: Dockerfile
    image: myimage
    links:
        - rabbitmq
    restart: always
    command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
    healthcheck:
        test: celery -b amqp://rabbitmq:5672 inspect ping -d celery@$$HOSTNAME
        interval: 30s
        timeout: 10s
        retries: 3

Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.

Ideally, rabbitmq should also have its own health check, perhaps with curl guest:guest@localhost:15672/api/overview, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.

Answer 3

For celery 5.2.3 I used celery -A [celery app name] status for the health check. This is how my docker-compose file looks like

worker:
      build: .
      healthcheck:
        test: celery -A app.celery_app status
        interval: 10s
        timeout: 10s
        retries: 10
      volumes:
          - ./app:/app
      depends_on:
        - broker
        - redis
        - database

Answer 4

Landed on this question looking for a health check for Celery workers as part of an Airflow setup (Airflow 2.3.4, Celery 5.2.7), which I eventually figured out. This is a very specific use case of the original question, but might still be useful for some:

# docker-compose.yml

  worker:
      image: ...
      hostname: local-worker
      entrypoint: airflow celery worker
      ...
      healthcheck:
        test: [ "CMD-SHELL", 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"' ]
        interval: 5s
        timeout: 10s
        retries: 10
      restart: always
      ...

I got inspiration from Airflow's quick-start Docker Compose.

Answer 5

As simple as that:

...
healthcheck:
    test: sh -c 'celery -A your_celery_module inspect ping'
    ...

Your log file will have a similar ping log:

[2023-05-05 11:18:47,309: DEBUG/MainProcess] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '8195241f-122a-3dc4-9841-739f55804b82'} ticket:bada7609-ce6b-499a-b3c2-c80911e0fe09]

Also, you might want to check it yourself in an already-running container. In that case, it will look something like this:

Answer 6

To configure a HEALTHCHECK for a Docker container running Celery tasks and ensure another container depends on its health status:

In your Docker Compose file:

version: "3.8"
services:
  celery:
    # Celery container config
    healthcheck:
      test: celery inspect ping
      interval: 1m
      timeout: 10s
      retries: 10
      start_period: 1m

  dependent_container:
    # Dependent container config
    depends_on:
      celery:
        condition: service_healthy

This sets up a health check for the celery container using celery inspect ping.
The dependent_container depends on celery with condition: service_healthy, ensuring it starts only when celery is healthy.