Batch union of django querysets

Question

I'm trying to get a single queryset from the output of a recursive function and i am hitting a performance problem.

Basically it seems that the act of trying to combine the individual querysets is doubling the time it takes to process (I was expecting this because of the implementation) but i was wondering if i could do this more efficiently.

def intersect(self, list_of_querysets):
    combined = list_of_querysets[0]
    for queryset in list_of_querysets:
        combined = combined | queryset
    return [combined]


def _get_template_folders(self, template_folder_list):
    """
    :rtype : list
    """

    parents = []
    for template_folder in template_folder_list:
        if not TemplateFolder.objects.filter(pk=template_folder).exists():
            continue
        templates = TemplateFolder.objects.filter(pk=template_folder)
        for template in templates:

            parent_folders = self._get_template_folders([template.template_folder_parent_id])
            if parent_folders is not None:
                parents.extend(parent_folders)

        if templates is not None:
            parents.append(templates)
    if parents:
        return parents
    else:
        return None

template_folders_list = self.intersect(self._get_template_folders(template_folder_list))

score 1 · Answer 1 · answered Apr 24 '13 at 15:31

1

Without seeing your models it's hard to tell, but it seems that your TemplateFolder model is some sort of tree structure. If that's so, you should investigate using something like MPTT to store it in the database: that's an algorithm that annotates a description of the tree structure, in terms of left and right node values, onto each row, and so makes querying everything under a specific parent extremely efficient. There's a great Django implementation: django-mptt.

answered Apr 24 '13 at 15:31

Daniel Roseman

588,541
66
880
895

It is and i should have mentioned that get_template_folders is about as efficient as it can be with the limitations imposed upon myself. what i was looking to do was to combine it with intersect to speed up the deduplicating part of the code as get_template_folders puts out a lot of dupes. I would love to use something like django-mptt, but unfortunately we're very tightly coupled to another application which has set out our database tables. We're just interfacing with it... Thanks for having a look. – Jharwood Apr 24 '13 at 15:52

Batch union of django querysets

1 Answers1