1

I am developing some code where i need to save in database big amount of data. Response is taking about a minute or more. So I decided to move that code in celery task. I have view that calls celery task. Problem here is that you shouldn't be able to receive status 200 from that view if celery task is still running for the same record in database.

There i used select_for_update statement for row level lock like:

    site = self.get_object()

    try:
        with transaction.atomic():
            Site.objects.select_for_update(nowait=True).filter(pk=site.pk)
    except DatabaseError:
        raise ObjectLockedException

in view, and:

    with transaction.atomic():
        models.Site.objects.select_for_update(nowait=True).get(pk=site_id)
        ....

in task. As you see, the code is the same.

My expected result is that if celery task has started and acquired lock for specific Site with site_id but it is not finished yet, I should catch the DatabaseError in the specified view. This is not the case, select_for_update in view passes without any exception even the task for same Site is still running.

Another funny thing is that on line serializer.save(), below the first snippet of code, in the specified view, my code waits for task to finish processing.

Am I doing it wrong?

Nikola Jevremovic
  • 19
  • 1

1 Answers1

0

The with block is a context manager. When the flow of control leaves that block, the context is closed, i.e. the transaction ends. You are locking records but not operating on them.

Try moving your update inside the with block.

Chris Johnson
  • 20,650
  • 6
  • 81
  • 80
  • If I move serializer.save() in `with` block, the task will throw exception at select_for_update, it is like i get deadlock on specific row. Probably, the problem here is that I overrided the model.save() method and in there I am calling my task. So i assume that select_for_update in task is triggered before serializer.save() is finished. – Nikola Jevremovic Aug 21 '19 at 09:45
  • Even if I move call of the task in the view after `serializer.save()`, view is blocked on `serializer.save()` while previous task is not finished. Really strange for me. – Nikola Jevremovic Aug 21 '19 at 10:10
  • I wonder could it be that you have database queries that are mutually exclusive that run in BOTH view and the task. If that is the case there is high probability you may end up with a database lock. – DejanLekic Aug 21 '19 at 10:55
  • They are exclusive. In view (serializer), I am updating just one field of Site object. Task should populate database with related objects (objects that have foreign key of site_id). I just want that you can't call this view if task is still populating database with related objects. For some reason, serializer.save() is waiting for task to finish and then triggering task again. I'm really confused right now – Nikola Jevremovic Aug 21 '19 at 11:05
  • I actually just found something that may work, but still not sure what is happening. It seems that problem exists because i have `pre_save` method for the Site object. If i put `select_for_update` in `pre_save` method, this seems to work. Its like `pre_save` method removes lock from view. – Nikola Jevremovic Aug 21 '19 at 11:20