How to avoid repeated django prefetch related database hits

Question

When i fetch a client i want to also prefetch all related probes and probe channels, and when i fetch a probe i also want to fetch all related probe channels.

But it seems that when i get a client object, or view the admin, it runs the get_queryset methods from both the client and the probe manager. so it will prefetch the probechannels twice. This can be seen both in the number of sql queries in debug toolbar, and by adding print statements in the get_queryset methods. so i've got

class ClientManager(models.Manager):
    def get_queryset(self, *args, **kwargs):
        return super(ClientManager, self).get_queryset(*args, **kwargs).prefetch_related('probe_set__probechannel_set')

class Client(models.Model):
    objects = ClientManager()

class ProbeManager(models.Manager):
    def get_queryset(self, *args, **kwargs):
        return super(ProbeManager, self).get_queryset(*args, **kwargs).prefetch_related('probechannel_set')

class Probe(models.Model):
    Client = models.ForeignKey('Client')
    objects = ProbeManager()

class ProbeChannel(models.Model):
    Probe = models.ForeignKey('Probe')

Is there a way to avoid this? I have read about _base_managers being used for related objects, but the _base_manager here is just models.Manager. I'm using django 1.11 and this trendy new thing called python 2.7.

First I'd get on Python 3 and Django 2. Prefetching is one of the most inconsistant things I've found in Django. Make sure you dont filter the lookups. Also I never used _set. I just used the field names. — kagronick, Apr 21 '18 at 18:11
How are you measuring queries being fired? Django query logging or Django Debug Toolbar? If you're using the Debug toolbar, can you share a screenshot of the SQL tab? — rtindru, Apr 24 '18 at 06:29
I am using Django Debug Toolbar, yes. Thanks for taking the time to answer- sorry i didn't take a screenshot- the model i've described was rather simplified, and I haven't had the time to produce a simplified version of the model to do the test. I've worked out the answer to my problem anyway, about to add to this question.... — rbennell, May 03 '18 at 11:51

score 0 · Accepted Answer · answered May 03 '18 at 11:55

I'm still not sure if this is desired Django behaviour, but the prefetches are definitely firing twice. The answer i've worked out to avoid ProbeManager prefetches to be fired as well was simple- avoid using this manager. So in the Probe class I've added

objects_without_prefetch = models.Manager()

and then used a prefetch object Prefetch('probe_set',Probe.objects_without_prefetch.all()) instead of simply using 'probe_set' which was calling its own prefetch.

Thanks anyway those that took the time to think about this and answer!

How to avoid repeated django prefetch related database hits

1 Answers1