VSCode debug Celery worker

Question

After some frustrating days of running this, I'm needing to look at debugging a celery worker process in VSCode. This is following the suggested process in the Celery docs for creating a message handler, rather than pub/sub from the same application.

The celery.py file:

from __future__ import absolute_import, unicode_literals
import os
import json

from celery import Celery, bootsteps
from kombu import Consumer, Exchange, Queue

dataFeedQueue = Queue('statistical_forecasting', Exchange('forecasting_event_bus', 'direct', durable=False), 'DataFeedUpdatedIntegrationEvent')

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')

app = Celery('statistical_forecasting')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Not required yet as handler is within this file
#app.autodiscover_tasks()


class DataFeedUpdatedHandler(bootsteps.ConsumerStep):
    def get_consumers(self, channel):
        return [Consumer(channel, queues=[dataFeedQueue],    callbacks=[self.handle_message], accept=['json'])]


def handle_message(self, body, message):
    event = json.loads(body)

    # removed for brevity, but at present echo's message content with print

    message.ack()

app.steps['consumer'].add(DataFeedUpdatedHandler)

My abbreviated project structure is:

workspace -
    vscode -
        - launch.json
    config -
        __init__.py            
        settings -
            local.py
    venv -
        celery.exe
    statistical_forecasting -
        __init__.py
        celery.py
        farms -
            __init__.py
            handlers.py    # ultimately handler code should live here...

From terminal with venv enable I'm running celery -A statistical_forecasting worker -l info which does appear to succeed in setting up and running the basic message handler.

What I've tried thus far with VSCode is setting up the follow configuration in launch.json

{
    "version": "0.2.0",
    "configurations": [
    {
        "name": "Python: Celery",
        "type": "python",
        "request": "launch",
        "module": "celery",
        "console": "integratedTerminal",
        //"program": "${workspaceFolder}\\env\\Scripts\\celery.exe",
        "args": [
            "worker",
            "-A statistical_forecasting",
            "-l info",
            ]
        },
    ]
}

Unfortunately this just results in the following message:

Error:
Unable to load celery application.
The module  statistical_forecasting was not found.

Logically I can reason that the debug should be running celery from the workspace directory and that it should see the statistical_forecasting directory with a __init__.py technical making it a module?

I've tried other various ideas, like forcing the program in lauch.json setting virtual environment etc. but all with the same basic Error message returned.

The 'init.py' within statistical_forecasting contains the standard Django setup, that I'm not convinced that it's required as celery task is run outside of Django and I don't intend publish/receive from the Django application.

I think every 10th django+celery question on SO is about module not found... Which is typically solved by correctly setting PYTHONPATH or similar. These errors have nothing to do with celery, or django, but how you run the software... — DejanLekic, Aug 29 '19 at 10:29
guilty as charged :-( issue here is working out where it needs to be set... VSCode should be picking up the PYTHONPATH, DJANGO_SETTINGS_MODULE should theoretically be picked up in the code... The camelCasing in the examples should give away that I'm primarily a C# dev so please be kind :-) — Ross Halliday, Aug 29 '19 at 10:35
I would gladly help, but I do not use django, and I am in general avoiding django+celery questions as they quite often are about configuring django environment. If you are getting that module not found error, that typically means your Python interpreter is running in wrong directory. Sure setting PYTHONPATH to the folder where statistical_forecasting.py is may help, but it may also break some other code... — DejanLekic, Aug 29 '19 at 10:46
no worries, I'll try setting up the Paths differently in VsCode to see if it helps with discovering the module. Thanks for commenting though; Stack Overflow sometimes feels like shouting into a void with no one listening. — Ross Halliday, Aug 29 '19 at 10:53
@DejanLekic it was the formatting of the args! see answer below. — Ross Halliday, Aug 29 '19 at 12:30

Ross Halliday · Accepted Answer · 2019-08-29T12:57:04.057

For the benefit of anyone else trying to do this, here's my minimal config to test celery as a module

    {
        "name": "Python: Celery",
        "module": "celery",
        "console": "integratedTerminal",
        "args": [
            "worker",
            "--app=statistical_forecasting",
            "--loglevel=INFO",
        ],
    },

The key takeaway id how the args are formatted. The original was using the shortened versions that you would usually see when running form the command line, in tutorials for example.

Usually you would see celery -A statistical_forecasting worker -l info for the debugger to work you need the fuller version celery --app=statistical_forecasting worker --loglevel=INFO.

Reflecting the comments below it also works as:

    {
        "name": "Python: Celery",
        "module": "celery",
        "console": "integratedTerminal",
        "args": [
            "worker",
            "-A",
            "statistical_forecasting",
            "-l",
            "info",
        ],
    },

Out of interest the longer version was as follows, but this mainly just repeats what VsCode will set by default:

    {
        "name": "Python: Celery",
        "type": "python",
        "request": "launch",
        "module": "celery",
        "console": "integratedTerminal",
        "cwd": "${workspaceFolder}",
        "program": "${workspaceFolder}\\env\\Scripts\\celery.exe",
        "pythonPath": "${config:python.pythonPath}",
        "args": [
            "worker",
            "--app=statistical_forecasting",
            "--loglevel=INFO",
        ],
        "env":{
            "DJANGO_SETTINGS_MODULE": "config.settings.local",
        }
    },

I think the reason why --app=blabla works for you is because when you use the -A you need two separate elements in the args array. This should work too: "args": ["worker", "-A", "statistical_forecasting", "--loglevel=INFO"] — DejanLekic, Aug 29 '19 at 12:47
Of course!!!, its passing the args as an array. I updated the answer for this way too. — Ross Halliday, Aug 29 '19 at 12:59

VSCode debug Celery worker

1 Answers1