16

As Celery documentation states, already executing task will not be aborted by calling .revoke(), unless terminate=True is set. But that is not recommended, because it will kill the worker itself, which might have already started another task. Does that mean that there is no reliable, stable way to do that?

EDIT: celery.contrib.abortable doesn't suit me, because, as documentation states, it works only with database backends.

nicks
  • 2,161
  • 8
  • 49
  • 101
  • It depends on the task you are trying to terminate (network, file, database operations, etc.), please take a look at this example: http://stackoverflow.com/questions/37039941/celery-python-revoke In my case my main task invokes child processes which are not terminated unless I send a SIGKILL signal. – gogasca May 23 '16 at 23:36
  • Possible duplicate of [Stopping celery task gracefully](http://stackoverflow.com/questions/16493364/stopping-celery-task-gracefully) – Louis Jun 07 '16 at 09:19

2 Answers2

3

A running task is a running subprocess of the worker (when using prefork), this means that the only way to abort a task is to kill the subprocess that is running it.

You may try to experiment your own implementation of revoke event handling trying to figure out the subprocess ID and kill only that one, but honestly don't know if is worth and if it can really work.

I think that short answer is you can't.

Anyway killing the worker is needed sometimes, especially in initial phases of projects where you still need to dimension correctly the resources, just make sure you log somewhere the running tasks so you can reschedule them or just use CELERY_ACKS_LATE

Mauro Rocco
  • 4,980
  • 1
  • 26
  • 40
  • "Anyway killing the worker is not unusual" What *usual* condition would warrant the forcible termination of a worker on a production system? I've used Celery for a few years now and the *only* time a worker is forcibly terminated is when the site must be shut down for maintenance. – Louis May 26 '16 at 10:45
  • 1
    Depends on use case, I worker for a big community website back in the days and we got some issues during peak times. Workers was taking up to much memory and freezing. Off course this was due to the fact that we didn't expect that load and worker where not dimensioned correctly. But I can tell you that in one year happened couple of times, without considering total crash of machine due to other factors. – Mauro Rocco May 26 '16 at 11:20
  • I clarified by answer after Louis comment. – Mauro Rocco May 26 '16 at 11:23
1

You can send HUP signal instead of TERM which gracefully restarts child process without killing worker.

In [80]: import signal

In [81]: x = add.delay(1, 2)

In [82]: x.revoke(terminate=True, signal=signal.SIGHUP)
Chillar Anand
  • 27,936
  • 9
  • 119
  • 136
  • "Other than stopping then starting the worker to restart, you can also restart the worker using the HUP signal, but note that the worker will be responsible for restarting itself so this is prone to problems and is not recommended in production" [Source](http://docs.celeryproject.org/en/latest/userguide/workers.html) – Louis May 26 '16 at 16:35
  • @Louis If you have a worker with a concurrency of 4, it has 4 process running 4 tasks at a time. Now restarting it is a bad idea just to revoke a task, since it might interrupt other process. The above command will only restart single process which was running that task. – Chillar Anand May 28 '16 at 23:15