7

I am having a strange issue where some of my tasks are getting dropped after sent to broker. This happens for around 1 out of 10 tasks. I have checked that there is no old celery worker consuming the task.

I used database backed backend and flower to monitor the missing tasks, but the task_id returned after apply_async is not present in database or in flower. Its status always comes as pending.

Then I used celery signals to find out what's happening. I found that for the missing tasks, only before_task_publish and after_task_publish signals are fired. Post that there is no trace for this task.

These are my signals

@before_task_publish.connect
def before_task_publish_handler(sender=None, headers=None, body=None, **kwargs):
# information about task are located in headers for task messages
# using the task protocol version 2.
  logger.info("BEFORE TASK SENT  id:"+body['id'])



@after_task_publish.connect
def after_task_publish_handler(sender=None, headers=None, body=None, exchange=None, routing_key=None, **kwargs):
# information about task are located in headers for task messages
# using the task protocol version 2.
    logger.info("AFTER TASK SENT  id:"+body['id'])


@task_prerun.connect
def task_prerun_handler(sender=None, task_id=None, task=None, **kwargs):
  logger.info("TASK PRERUN with TASK_ID:"+str(task_id))

This is what I found in logs

$ cat gunicorn-access.log | grep -i 103de274-00dc-4765-844f-d319e9e199c2
  BEFORE TASK SENT id: '103de274-00dc-4765-844f-d319e9e199c2'
  AFTER TASK SENT  id: '103de274-00dc-4765-844f-d319e9e199c2'

I am not sure whether the task is ignored by rabbitmq or its silently dropped for some reason.

Anurag
  • 1,521
  • 2
  • 17
  • 34

1 Answers1

4

Once in a while celery loses tasks before it gets executed. If you don't want to lose them, you should set enable task_acks_late(CELERY_ACKS_LATE in old versions).

In your celery settings, set 

task_acks_late = True

This makes sure that the task messages will be acknowledged after the task has been executed.

Chillar Anand
  • 27,936
  • 9
  • 119
  • 136
  • that doesn't solve my problem. tcpdump shows me amqp packets are being sent to rabbitmq server but no packets received on other end. Client and server are on different Azure accounts. May be Azure's tcp connection timeout configuration affecting it but using keepalive or heartbeat didn't solve it either. – melih Jul 11 '17 at 13:02