Problem
I have a Django + Celery app, and am trying to get a lengthy method to run in a Celery Task.
Goal
I want to
1) Put an entire line of execution - that calls across multiple classes & files - into a Celery Task. The application should continue serving users in the meantime.
2) Signal back when the Celery Task is done so I can update the UI.
Issue
Despite scouring docs, examples, and help sites I can't get this working.
I don't know which signals to use, how to register them, how to write the methods involved to receive those signals properly, and respond back when done.
It isn't clear to me from the Celery documentation how to accomplish all of this: there are many, many signals and decorators available but little clarification on which ones are appropriate for this sequence of events.
Help Requested
Either
- A point in the direction of some resources that can help (better documentation, links to examples that fit my use case, etc...),
or
- Some first-hand help with which signals are appropriate to use, where to use them, and why.
This is my first post on StackOverflow, so thanks in advance for your kindness and patience.
The desired execution path:
1) views.py: Get input via GET or POST
SomeMethod: Send Signal to staticmethod in TaskManager (housed in TaskManager.py) to start a task with arguments
2) TaskManager.py: Process Signals, Keep Track of Tasks Running
SignalProcessor: receive signal, pull arguments out of kwargs, call appropriate method in tasks.py.
TaskList: Make note of the task in a TaskList so I know it's running, and where.
3) tasks.py
@shared_task HandleTheTask(arg1, arg2):
Call the appropriate sequence of methods in other files/classes in sequence, which ultimately writes to the database many times in many methods. These other methods are not inside tasks.py.
Send a Signal to TaskManager when the method in tasks.py completes the lengthy task.
4) TaskManager.py
SignalProcessor: receive task_complete signal, remove the appropriate task from the TaskList.
What I've already tried
- Putting HandleTheTask(arg1, arg2) with a @task or @shared_task decorator into tasks.py.
I've tried calling HandleTheTask.delay(arg1, arg2) from the SignalProcessor method.
Result: Nothing happens. The app continues to execute, but the lengthy task doesn't. Celery doesn't notice the call and never executes it.
This is probably because the method I want to run is in another process?
- Putting an intermediary method into tasks.py, which calls back to TaskManager.py to run the lengthy task.
Result: As above.
- Looking up Celery Signals to properly signal across processes: This looks like the right solution but the documentation is big on info, small on guidance.
Resources I've Consulted
I've already looked a bunch of docs and help sites. Here are just a few of the resources I've consulted:
- https://docs.celeryproject.org/en/stable/userguide/signals.html
- https://medium.com/analytics-vidhya/integrating-django-signals-and-celery-cb2876ebd494
- How will I know if celery background process is successful inside django code. If it is successful I want render a html page
- Catch django signal sent from celery task
- Django Signals in celery
Any help is welcome. Thank you!
