django-dramatiq pipelines only run first stage

Question

I am trying to use django-dramatiq to run a pipeline of several stages, each defined as dramatiq Actors, using the pipeline(<stages>).run() method but it's only running the first stage/Actor and not attempting the other stages.

I've defined some cutdown fake actors to illustrate the problem:

import dramatiq

@dramatiq.actor
def fake_extract(process_pk, *args, **kwargs):
    print(f"fake_extract: Process PK= {process_pk} Running extract on {kwargs['fits_file']}")

@dramatiq.actor
def fake_astromfit(process_pk, *args, **kwargs):
    print(f"fake_astromfit: Process PK= {process_pk} Astrometric fit on {kwargs['ldac_catalog']}, updating {kwargs['fits_file']}")

@dramatiq.actor
def fake_zeropoint(process_pk, *args, **kwargs):
    print(f"fake_zeropoint: Process PK= {process_pk} ZP determination on {kwargs['ldac_catalog']} with {kwargs['desired_catalog']} ref catalog")

I've then defined the stages and built a pipeline:

import os
from dramatiq import pipeline
from test_dramatiq.dramatiq_tests import fake_extract, fake_astromfit, fake_zeropoint


fits_filepath = '/foo/bar.fits'
fits_file = os.path.basename(fits_filepath)

steps = [{
            'name'   : 'proc-extract',
            'runner' : fake_extract,
            'inputs' : {'fits_file':fits_filepath,
                       'datadir': os.path.join(dataroot, temp_dir)}
        },
        {
            'name'   : 'proc-astromfit',
            'runner' : fake_astromfit,
            'inputs' : {'fits_file' : fits_filepath,
                        'ldac_catalog' : os.path.join(dataroot, temp_dir, fits_file.replace('e91.fits', 'e91_ldac.fits')),
                        'datadir' : os.path.join(dataroot, temp_dir)
                        }
        },
        {
            'name'   : 'proc-zeropoint',
            'runner' : fake_zeropoint,
            'inputs' : {'ldac_catalog' : os.path.join(dataroot, temp_dir, fits_file.replace('e91.fits', 'e92_ldac.fits')),
                        'datadir' : os.path.join(dataroot, temp_dir),
                        'desired_catalog' : 'PS1'
                        }
        }]

pipes = []
for step_num, step in enumerate(steps):
    inputs = step['inputs']
    print(f"  Performing pipeline step {step['name']}")
    pk = 1234+step_num
    pipes.append(step['runner'].message_with_options(args=[pk,], kwargs=inputs, pipe_ignore=True))
pipeline(pipes).run()

Running inside ipython with regular dramatiq these seems to work fine and all stages run:

fake_extract: Process PK= 1234 Running extract on /foo/bar.fits
fake_astromfit: Process PK= 1235 Astrometric fit on /foo/Temp_cvc2/bar.fits, updating /foo/bar.fits
fake_zeropoint: Process PK= 1236 ZP determination on /foo/Temp_cvc2/bar.fits with PS1 ref catalog

However defining them in a module imported by django-dramatiq via the Django project's settings.py file and defining the pipeline as above in the python manage.py shell and with the python manage.py rundramatiq runner only runs the first stage/Actor:

fake_extract: Process PK= 1234 Running extract on /foo/bar.fits

and it never executes the other stages... Any ideas on what is going on here and why multi-stage pipelines are not working under django-dramatiq ?

score 0 · Answer 1 · answered Aug 24 '22 at 23:37

So it turns out the problem was due to a lack of missing middleware in the Django project's settings.py which was causing it to silently partly work. I had:

DRAMATIQ_BROKER = {
    'BROKER': 'dramatiq.brokers.redis.RedisBroker',
    'OPTIONS': {
        'url': f'redis://{REDIS_HOSTNAME}:6379',
    },
    'MIDDLEWARE': [
        'dramatiq.middleware.AgeLimit',
        'dramatiq.middleware.TimeLimit',
        'dramatiq.middleware.Callbacks',
        'dramatiq.middleware.Retries',
        'django_dramatiq.middleware.DbConnectionsMiddleware',
    ]
}

but the MIDDLEWARE was missing a 'dramatiq.middleware.Pipelines', in there as well. Adding that in and restarting the runserver and the rundramatiq management commands caused both the slimmed-down test example above and the original fully featured version to start working.

django-dramatiq pipelines only run first stage

1 Answers1