0

As the subject says, tasks that used to retry until they reach defined "max_retries" count, now sometimes stop doing that after random number of times :/. Sometimes they stop retrying after couple hundred times, sometimes after just few times.

What I noticed is that if I restart "celery beat" process - after some minutes, some tasks, that "were quite" for hours, start retrying again as they should.

Can't pinpoint precisely when it started happening, but it might be after upgrade of Celery, RabbitMQ or Django.

Anybody have an idea why this is happening ?

I'm running:

Django: 1.7.3
RabbitMQ: 3.4.2
celery:3.1.17
kombu:3.0.24
billiard:3.3.0.19
python:2.7.3
py-amqp:1.4.6

Celery settings:

CELERY_ACKS_LATE            = True
CELERY_SEND_EVENTS          = True
CELERY_TRACK_STARTED        = True
CELERY_DISABLE_RATE_LIMITS  = True
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_SEND_TASK_SENT_EVENT = True

Task code looks smth. like this:

class ABCTask(AbortableTask):
    ignore_result = False
    max_retries = 288*5

    def run(self):
        try:
            [...]
        except NoAvailableDevices as e:
            try:
                self.retry(exc=e)
            except MaxRetriesExceededError, e:
                [...]

Thanks, Lauris

Lauris
  • 13
  • 2
  • Where exactly are you setting max retries? – user2097159 Jan 21 '15 at 15:31
  • Have a look at the code in the end of the main post. – Lauris Jan 21 '15 at 15:48
  • I see where you set `max_retries = 288*5` but I don't see you doing `self.retry(exc=e, max_retries=max_retries)` – user2097159 Jan 21 '15 at 15:52
  • You don't see me doing that because I don't want to override default retry value, which I still need to be 288*5. The issue here is the fact that the task sometimes stops retrying before it tries 288*5 times. – Lauris Jan 21 '15 at 22:11
  • I think I know the reason tasks were not being retried - all my workers were busy with other tasks, so there were no free workers available. – Lauris Jan 26 '15 at 14:40

0 Answers0