5

I want to run a complex task scheduled by beat. Let us assume the default add/mul tasks are defined.

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    sender.add_periodic_task(
        crontab(),
        add.s(2,3) | mul.s(2)
    )

But this will return an error in the worker:

NotImplementedError: chain is not a real task

How can I schedule a non trivial task with celery beat?

Serge3
  • 53
  • 1
  • 5
  • I suggest you use beat configuration and schedule your task to run however/whenever you like there... Also, use beat as a separate service... – DejanLekic Jun 05 '19 at 15:43
  • I have used a beat schedule (as given above). But the task has to depend on the previous result and is therefore a chain task. However celery beat FAILS when i provide it a chain task with the error given above. – Serge3 Jun 05 '19 at 16:30
  • 3
    I would wrap that chain in a task, and add it to the beatconfig... – DejanLekic Jun 05 '19 at 16:42
  • 1
    as recommended by @DejanLekic wrapping the chain (or a group or chains in my case) in a task worked for me. The only extra thing I had to do was to actually call the group with `.delay()` inside the task definition. Without that, the main task was scheduled fine but the linked tasks where never triggered. Tested with Celery 5.1.2 – howaryoo Aug 10 '21 at 07:47

2 Answers2

6

One way to do that is to schedule your tasks chain in beat_schedule in your celeryconfig, using link option, celery_tasks here is a module name where your tasks are defined

from celery.schedules import crontab
from celery import signature

beat_schedule = {
    'chained': {
        'task': 'celery_tasks.add',
        'schedule': crontab(),
        'options': {
            'queue': 'default',
            'link': signature('celery_tasks.mul',
                        args=(),
                        kwargs={},
                        options={
                            'link': signature('celery_tasks.another_task', 
                                args=(),
                                kwargs={}, 
                                queue='default')
                        },
                        queue='default')
            },
         'args': ()
    }
}
Greenev
  • 871
  • 6
  • 23
  • I have tried this now. The chained task is accepted, but not executed by the worker., The beat service states ```[2019-06-06 13:22:00,034: INFO/MainProcess] Scheduler: Sending due task chained (tasks.add)``` but the worker does not react at all. Additionally, can I link more then one task? I eventually will need 4 linked tasks (object loader, dmap (distribute to group), load_data, process_data). – Serge3 Jun 06 '19 at 11:28
  • Is your worker listen to the queue you've specified in `beat_schedule`? – Greenev Jun 06 '19 at 11:54
  • I am only editing a single file `tasks.py`. I have done more experiments. I have added the task schedule as above (see code). Only the `tasks.add` task is executed. //edit: I forgot there was the second "queue" argument in the link-dict. For the add|mul task it seems to work now. I will have to test with my more complex schedule next. Thanks. – Serge3 Jun 06 '19 at 12:07
  • To run more linked tasks you can add `link` attribute to `options` in your previous linked task. I've updated my answer to demonstrate this – Greenev Jun 06 '19 at 12:11
  • You are welcome! Note, that I ended up using just `Cron` to run periodic tasks in production as I couldn't force celery-beat to work properly, may be it was fixed in the latest versions of Celery though – Greenev Jun 06 '19 at 12:16
  • According to the latest documentation (5.1.2): `In practice the link execution option is considered an internal primitive, and you’ll probably not use it directly, but use chains instead.` [Linking (callbacks/errbacks)](https://docs.celeryproject.org/en/stable/userguide/calling.html#linking-callbacks-errbacks) – howaryoo Aug 10 '21 at 07:59
  • 1
    Good catch! The original solution was devised for celery 4.x, so there could be a better one for the newest version – Greenev Aug 10 '21 at 09:44
0

for add chained periodic tasks you can use an @app.task when declare your chain and then, add this new task on add_periodic_task() method. Example:

@app.on_after_finalize.connect ->i use this because it`s declared on task.py
def setup_periodic_tasks(sender, **kwargs):                               
   sender.add_periodic_task(timedelta(minutes=10), chian_st22.s(),name='test')
                                

@app.task
def chian_st22(): -> i create the task with chain 
    cadena = chain(st22.s(), mailer.s()).apply_async()

@app.task
def mailer(data):
    clase = CheckAlert()
    mail = clase.envio_mail(data)
    return mail

@app.task
def st22():
    clase = CheckAlert()
    st = clase.check_st22_dumps()
    return st
enzo rossi
  • 1