How to soft delete many to many relation with Django

Question

In my Django project, all entities deleted by the user must be soft deleted by setting the current datetime to deleted_at property. My model looks like this: Trip <-> TripDestination <-> Destination (many to many relation). In other words, a Trip can have multiple destinations.

When I delete a Trip, the SoftDeleteManager filters out all the deleted trip. However, if I request all the destinations of a trip (using get_object_or_404(Trip, pk = id)), I also get the deleted ones (i.e. TripDestination models with deleted_at == null OR deleted_at != null). I really don't understand why since all my models inherit from LifeTimeTracking and are using the SoftDeleteManager.

Can someone please help me to understand why the SoftDeleteManager isn't working for n:m relation?

class SoftDeleteManager(models.Manager):
    def get_query_set(self):
        query_set = super(SoftDeleteManager, self).get_query_set()
        return query_set.filter(deleted_at__isnull = True)

class LifeTimeTrackingModel(models.Model):
    created_at = models.DateTimeField(auto_now_add = True)
    updated_at = models.DateTimeField(auto_now = True)
    deleted_at = models.DateTimeField(null = True)

    objects = SoftDeleteManager()
    all_objects = models.Manager()

    class Meta:
        abstract = True

class Destination(LifeTimeTrackingModel):
    city_name = models.CharField(max_length = 45)

class Trip(LifeTimeTrackingModel):
    name = models.CharField(max_length = 250)
    destinations = models.ManyToManyField(Destination, through = 'TripDestination')

class TripDestination(LifeTimeTrackingModel):
    trip = models.ForeignKey(Trip)
    destination = models.ForeignKey(Destination)

Resolution I filed the bug 17746 in Django Bug DB. Thanks to Caspar for his help on this.

Answer 1

To enable filtering by deleted_at field of Destinantion and Trip models setting use_for_related_fields = True for SoftDeleteManager class is enough. As per Caspar's answer this does not return deleted Destinations for trip_object.destinations.all().

However from your comments we can see you would like to filter out Destinations that are linked to Trip via a TripDestination object with a set deleted_at field, a.k.a. soft delete on a through instance.

Let's clarify the way managers work. Related managers are the managers of the remote model, not of a through model.

trip_object.destinantions.some_method() calls default Destination manager. destinantion_object.trip_set.some_method() calls default Trip manager. TripDestination manager is not called at any time.

You can call it with trip_object.destinantions.through.objects.some_method(), if you really want to. Now, what I would do is add an Instance method Trip.get_destinations and a similar Destination.get_trips that filters out deleted connections.

If you insist on using the manager to do the filtering it gets more complicated:

class DestinationManager(models.Manager):
    use_for_related_fields = True

    def get_query_set(self):
        query_set = super(DestinationManager, self).get_query_set()
        if hasattr(self, "through"):
            through_objects = self.through.objects.filter(
                destination_id=query_set.filter(**self.core_filters).get().id,
                trip_id=self._fk_val,
                deleted_at__isnull=True)
            query_set = query_set.filter(
                id__in=through_objects.values("destination_id"))

        return query_set.filter(deleted_at__isnull = True)

The same would have to be done for TripManager as they would differ. You may check the performance and look at django/db/models/fields/related.py for reference.

Modifying the get_queryset method of the default manager may hamper the ability to backup the database and the documentation discourages it. Writing a Trip.get_destinations method is the alternative.

Answer 2

It looks like this behaviour comes from the ManyToManyField choosing to use its own manager, which the Related objects reference mentions, because when I try making up some of my own instances & try soft-deleting them using your model code (via the manage.py shell) everything works as intended.

Unfortunately it doesn't mention how you can override the model manager. I spent about 15 minutes searching through the ManyToManyField source but haven't tracked down where it instantiates its manager (looking in django/db/models/fields/related.py).

To get the behaviour you are after, you should specify use_for_related_fields = True on your SoftDeleteManager class as specified by the documentation on controlling automatic managers:

class SoftDeleteManager(models.Manager):
    use_for_related_fields = True

    def get_query_set(self):
        query_set = super(SoftDeleteManager, self).get_query_set()
        return query_set.filter(deleted_at__isnull = True)

This works as expected: I'm able to define a Trip with 2 Destinations, each through a TripDestination, and if I set a Destination's deleted_at value to datetime.datetime.now() then that Destination no longer appears in the list given by mytrip.destinations.all(), which is what you are after near as I can tell.

However, the docs also specifically say do not filter the query set by overriding get_query_set() on a manager used for related fields, so if you run into problems later, bear this in mind as a possible cause.