Django: write something to the database in a catch block, when using an atomic transaction

Question

I have a Django REST Framework serializer which uses select_for_update in combination with atomic transitions, like this: https://docs.djangoproject.com/en/4.2/ref/models/querysets/#select-for-update/. That works fine, except that I want to write something to the database when an error is thrown... and these insert statements are getting rolled back, never making it to the database.

The code is something like this (very much simplified but gets the point across and is reproducible):

models.py

from django.db import models


class LicenseCode(models.Model):
    code = models.CharField(max_length=200, unique=True)


class Activation(models.Model):
    license_code = models.TextField(max_length=200)
    activation_log = models.TextField(blank=True, null=True)
    success = models.BooleanField(default=False)

views.py

from django.http import HttpResponse, Http404
from django.db import transaction
from .models import Activation, LicenseCode

class LicenseRedeemSerializer:
    @transaction.atomic
    def save(self):
        license_codes = LicenseCode.objects.all().select_for_update()

        activations = []

        for license_code in license_codes:
            activations.append(
                Activation(license_code=license_code.code, success=False)
            )

        self.activate_licenses(activations)

    def activate_licenses(self, activations):
        try:
            # In our real app we'd try to activate the licenses with an external SDK. This can fail.
            raise Exception("Something went wrong!")

        except Exception as e:
            for activation in activations:
                activation.activation_log = str(e)
                activation.save()

            # With our real DRF serializer we'd raise ValidationError
            raise Http404("Could not activate the license!")


def view(request):
    # Let's make sure we have a license code to work with
    LicenseCode.objects.get_or_create(code="A")

    serialier = LicenseRedeemSerializer()
    serialier.save()

    html = "Hello there"
    return HttpResponse(html)

The problem I am facing is that when the external SDK triggers an error and I'm trying to write something to the database, that this never ends up in the database, the transaction is just rolled back.

How can I make sure that I can still write something to the database when using atomic transactions, in the except block?

ruddra · Answer 1 · 2023-04-11T11:35:21.313

As atomic() documentation mentions, it will roll back the transaction on exception, thus I do not think there is a way to store the information directly in Database during error.

But you can always catch the exception outside of the atomic block, then save the error and re-raise the error like this:

class MySerializer(Serializer):
    # Some serializer fields here..


    def save(self):
       try:
           self.save_data()
       except ValidationError as e:
           Log.objects.create(error = str(e))
           raise e

    @transaction.atomic
    def save_data(self):
        foo = Foo.objects.all().select_for_update()
        self.do_something(foo)
        self.do_something_else(foo)

    def do_something(self, foo):
        try:
           SomeExternalSDK.doStuff(foo)
        except SomeExternalSDK.SomeException as e:
           raise ValidationError(str(e))

    def self.do_something_else(self, foo):
        pass

Alternatively you can create a object variable (like a list) where you can put the exceptions happening in the SDK and then later store those errors in DB.

Usually, logs are not stored in DB, rather stored in the files. Instead of storing the errors in DB, you can consider storing errors in file system or use Django's logging to store the errors.

Update

You can remove the nested exception raising which is allowing the rollback happening. Instead, you can use a flag to raise 404 later.

@transaction.atomic
def save(self):
    license_codes = LicenseCode.objects.all().select_for_update()
    activations = []
    for license_code in license_codes:
        activations.append(
            Activation(license_code=license_code.code, success=False)
        )

    error = self.activate_licenses(activations)  # using flag to see if there is any error
    return error

def activate_licenses(self, activations):
    has_error = False
    try:
        raise Exception("Something went wrong!")

    except Exception as e:
        for activation in activations:
            activation.activation_log = str(e)
            activation.save()
        # flag
        has_error = True
    return has_error



 #view
   def view(request):
    # Let's make sure we have a license code to work with
    LicenseCode.objects.get_or_create(code="A")

    serialier = LicenseRedeemSerializer()
    error = serialier.save()
    if error:
       raise Http404('error activation')

Alternatively, you can look into savepoint but I am not sure how it will be applicable in your code.

The problem with this is that you're still raising an error, so the view using the serializer is still the one rolling back the transaction.. because an error happened. — Kevin Renskers, Apr 11 '23 at 10:01
No. I've updated the example code in my question again, to make it very specific and reproducible. — Kevin Renskers, Apr 11 '23 at 10:35

patriotyk · Accepted Answer · 2023-04-11T13:13:14.137

1

Instead of decorator you can use context manager. For example:

from django.http import HttpResponse, Http404
from django.db import transaction
from .models import Activation, LicenseCode

class LicenseRedeemSerializer:
    def save(self):
        license_codes = LicenseCode.objects.all().select_for_update()

        activations = []

        with transaction.atomic():
            for license_code in license_codes:
                activations.append(
                    Activation(license_code=license_code.code, success=False)
                )

        self.activate_licenses(activations)

    def activate_licenses(self, activations):
        try:
            # In our real app we'd try to activate the licenses with an external SDK. This can fail.
            raise Exception("Something went wrong!")

        except Exception as e:
            for activation in activations:
                activation.activation_log = str(e)
                activation.save()

            # With our real DRF serializer we'd raise ValidationError
            raise Http404("Could not activate the license!")


def view(request):
    # Let's make sure we have a license code to work with
    LicenseCode.objects.get_or_create(code="A")

    serialier = LicenseRedeemSerializer()
    serialier.save()

    html = "Hello there"
    return HttpResponse(html)

edited Apr 11 '23 at 13:13

answered Apr 09 '23 at 20:44

patriotyk

497
3
13

I think this doesn’t work because of select_for_update, which needs to be in an atomic transaction. But I’ll try in the morning. – Kevin Renskers Apr 09 '23 at 21:54
It should work. Documentation shows in example how to use it, and now it is exactly the same as in the documentation. – patriotyk Apr 10 '23 at 07:46
I am getting a fatal error though: `select_for_update cannot be used outside of a transaction.` – Kevin Renskers Apr 10 '23 at 10:44
Hm, you can put it inside transaction. Strange that documentation states another example. I have updated my code. – patriotyk Apr 10 '23 at 13:24
Sadly I can't do this in my real-world app, as the code that is fetching data from the database isn't immediately next to the code that can catch exceptions. I can't put `with transaction.atomic()` inside of the `try` block. So what I really need to be able to do is that the `except` block can "break out" of the transaction, so that its queries won't be rolled back. – Kevin Renskers Apr 10 '23 at 16:17
I've updated my example in the question to be more similar to the real world code – Kevin Renskers Apr 10 '23 at 16:19
Looks like line `foo = Foo.objects.all().select_for_update()` in your code is different. Are you sure you are not evaluating this queryset in that line? maybe you are doing some transformations using list comprehension or evaluate it in some another way? – patriotyk Apr 10 '23 at 18:53
Yes, the queryset is evaluated before sending it to do_something: it's looped over and things are done with those objects. – Kevin Renskers Apr 11 '23 at 09:36
Ha, that explains everything. I have updated my answer because I thing it is more correct than accepted one. – patriotyk Apr 11 '23 at 13:12
Sadly your example doesn't work in my real-world app because the license codes are actually sent along to the `activate_licenses` method, and it's that code which should be able to write to the database whenever there's an error. It's hard to know how complex to make these repro examples – Kevin Renskers Apr 11 '23 at 17:32
1

Never mind - I was able to refactor my code and your fix works! I can give you a bounty in 23 hours. – Kevin Renskers Apr 11 '23 at 17:37

Django: write something to the database in a catch block, when using an atomic transaction

models.py

views.py

2 Answers2

Update