Python / Celery : how can I kill subtasks when killing a parent task?

Question

Context

I've created a Django application that is calling a celery task which in turns spawn other tasks and wait for them to be finished.

Here's the workflow :

1) The main python/django code start a celery task in the background

2) The celery task process some code and then start a celery group of differents tasks and wait for them to be ready

3) every task of the group then spawn another group of subtasks in the same way and wait for them to finish

It works well (although I'm a begginer and probably implemented it poorly) but now I would like to be able to terminate every child processes if I kill the main celery tasks started in the beggining.

What I have so far

I've recreated the situation using a simple parent tasks spawning multiple child tasks and i've modified the "on_failure" method of the celery Task class to kill it's child when it fails.

Tasks.py

from celery import Celery, group,Task, result
from celery.signals import task_revoked
import time
from pprint import pprint
application = Celery('tasks',backend='amqp://',broker='amqp://guest@localhost//')


class MyTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        print(self.AsyncResult(task_id).children[0].revoke(terminate=True,signal='SIGTERM'))
        print('{0!r} failed: {1!r}'.format(task_id, exc))

@application.task(base=MyTask)
def childTask():
    while True:
        time.sleep(10)
        print("Message de la tache enfant")
        continue

@application.task(base=MyTask)
def parentTask(pra_id = None):
    child_tasks = []
    print("Lancement tache mère")
    child_tasks.append(childTask.s())
    child_tasks.append(childTask.s())
    child_tasks.append(childTask.s())
    tasks = group(child_tasks)
    tasks.apply_async()

    time.sleep(15)
    raise KeyError

main.py

from tasks import parentTask

parent1 = parentTask.delay(pra_id = 10)
parent2 = parentTask.delay(pra_id = 20)

When the code raise an error, the parent task gets killed succesfully and its child tasks too, this is what I want.

What I need

I need to be able to manually kill my parent task from my django application.

This is done by inspecting the celery worker and finding my task by searching for it's arguments, this was successfully done, however, When I manually revoke the celery task once I find it, it does not terminate the child tasks spawned by this task and that's what I need.

What I've tried so far

I've tried to create a function triggered by the "revoke" signal

(http://docs.celeryproject.org/en/latest/userguide/signals.html#task-revoked)

that would execute when a task is revoked.

Capturing the signal worked (I was able to execute some code when revoking a task) but I was not able to use the same code as my "On_failure" method described above) to kill the childs tasks.

Problem

The Request object sent to the function does contain my parent Task but the "children" property of the class is empty when it should contain a GroupResult object containing the child tasks.

score 1 · Answer 1 · answered Oct 02 '19 at 15:07

Not sure if this is helpful for you, but what I've found works somewhat ok is to store each subtask id in either redis or some database as they are created and associate them with a pipeline_id. Then if I need to kill the parent task, I can also kill all of the subtasks that were stored in the list as well.

result.revoke(terminate=True)

subtask_results = get_subtask_status(pipeline_id) #Custom Function

for subtask_result in subtask_results:
    subtask_result.revoke(terminate=True)

score 1 · Answer 2 · answered Oct 02 '19 at 15:28

By default Celery Task object has trail = True which means it will store its children. So you will be able to get it using request.children, or using (Async)Result's children attribute. Once you have list of child task_ids it is trivial to revoke those tasks.

Keep in mind that in some cases Celery will not be able to revoke a task even with terminate=True, so you may actually need to send it a SIGKILL by calling revoke(terminate=True, signal='SIGKILL'). This is not a bug in Celery, but it more/less depends on the nature of the task and what does it do...

Python / Celery : how can I kill subtasks when killing a parent task?

2 Answers2