Celery is not consuming with two tasks together

Question

I have celery snippets as follows:

first define the Task Class in celery_tasks.tasks.py

import celery


class LoggerDefine(celery.Task):
    name = 'message-logger'

    def run(self, payload):
        pass


class PerformanceMeasureDefine(celery.Task):
    name = 'performance-logger'

    def run(self, payload, elapseTime):):
        pass

then implement the methods in from consumers.logger

import django
django.setup()
from log_models.models import Level_logs
from django.conf import settings
from common_functions.logger import *
from celery_tasks.tasks import LoggerDefine, PerformanceMeasureDefine



class LoggerImpl(LoggerDefine):
    def run(self, payload):
        fmt1 = 'Message body: {}'
        print("-------------------------------- log persistor consuming --------------------------------")
        Level_logs.objects.create(
            level=payload.get(LEVEL),
            date_time=payload.get(DATETIME),
            file_name=payload.get(FILENAME),
            line_number=payload.get(LINE_NUMBER),
            function_name=payload.get(FUNCTION),
            payload=payload.get(PAYLOAD),
        )
        return fmt1.format(payload)




class PerformanceMeasureImpl(PerformanceMeasureDefine):
    def run(self, payload, elapseTime):
        fmt1 = 'Message body: {}'
        print("-------------------------------- performance consuming --------------------------------")
        print("payload = ", payload)
        print("elapseTime = ", elapseTime)
        return fmt1.format(payload)

a function in celery_tasks.utils to create celery workers and consumers

import celery

def create_worker_from(WorkerClass):
    assert issubclass(WorkerClass, celery.Task)
    app = celery.Celery()
    app.config_from_object("django.conf:settings")
    app.conf.update(task_default_queue=WorkerClass.name) # update worker queue
    worker_task = app.register_task(WorkerClass())

    return app, worker_task

where the celery setting put in django setting.py file is:

CELERY_BROKER_URL = "amqp://guest:guest@rabbitmq:5672//"

finally, I instantiate them as two consumers

from consumers.logger import LoggerImpl, PerformanceMeasureImpl
from celery_tasks.utils import create_worker_from

logger, _ = create_worker_from(LoggerImpl)
logger.worker_main(["worker", "-c", "2"])

performance_measure, _ = create_worker_from(PerformanceMeasureImpl)
performance_measure.worker_main(["worker", "-c", "2"])

then it is not working, I tried to comment out performance_measure and leave only logger like:

from consumers.logger import LoggerImpl, PerformanceMeasureImpl
from celery_tasks.utils import create_worker_from

logger, _ = create_worker_from(LoggerImpl)
logger.worker_main(["worker", "-c", "2"])

#performance_measure, _ = create_worker_from(PerformanceMeasureImpl)
#performance_measure.worker_main(["worker", "-c", "2"])

then logger works properly. I also tried comment out logger and leave only performance_measure like:

from consumers.logger import LoggerImpl, PerformanceMeasureImpl
from celery_tasks.utils import create_worker_from

#logger, _ = create_worker_from(LoggerImpl)
#logger.worker_main(["worker", "-c", "2"])

performance_measure, _ = create_worker_from(PerformanceMeasureImpl)
performance_measure.worker_main(["worker", "-c", "2"])

then performance_measure works properly.

so only one consumer is working properly fine, but two together are not.

There is an error and message stucked in message queue without consuming when two of them are working together.

consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: [Errno 111] Connection refused.

by the way, the publisher usage is like

_, logger_worker = create_worker_from(LoggerDefine)
_, performance_measure_worker = create_worker_from(PerformanceMeasureDefine)

logger_worker.apply_async(args=[msg,])
performance_measure_worker.apply_async(
                args=[request.data, elapsed_time])

How can I make two of them work?

Thanks a lot ahead.

Answer 1

I am not familiar with django much but one celery consumer can only be started from one worker class. To get a better understanding of celery, start the worker in --loglevel DEBUG mode.

An overview of Celery terminology and their meanings:

1. worker: Long running process that listens on the broker, fetches messages as they come from the broker and executes them in the current process or its child process(number of child processes can be specified with concurrency)
2. task: Object with unique name(usually import name) that celery worker registers in its registry. After receiving a message from the broker celery first checks whether the task unique string is present in its registry and assigns a worker to process it.
3. queue: Queue is a channel on broker. While sending tasks, queue can be specified (defaults to default) and message is stored on that channel on broker. Multiple queues can be used to deal with different priorities and workloads. Each worker can also listen on multiple queues and process messages as they come.
4. broker: Broker stores messages until they are processed by any of the workers. Different configurations can be set for different brokers. Client/Producer serialises the task and sends those messages to the broker on the defined queue. Messages are stored on the broker until they are picked up by any of the workers listening on that queue.

Any number of producers and consumers can be added without the knowledge of each other as it's the broker that does the job of storing information about the active queues, workers and producers.

It's recommended to use only one instance of Celery if only single broker is used. Here, two objects were instantiated while each worker expects only one such class. You can either create two files each for logger and performance and pass them on the command line while starting the worker. Or you can conditionally configure the queue and register tasks based on a command line argument while starting the worker. Either ways, you have to run two workers for this to work.