Performance impact of or-ing Q objects in django query

Question

I am performing a query that is or-ing a bunch Q's together and it seems to be taking a lot of time. Here is some psuedo code

query_params = []
for i in range(80): #there are about 80ish Q objects being created
    query_params.append(Q(filter_stuff))

Then I or them all together

query_params = reduce(or_, query_params)

And when I execute the query

query = list(MyModel.objects.filter(query_params))

It hangs for a LONG time. I know this is a pretty general question and it's hard to given a diagnostic without an intimate understanding of the data structure (which would be difficult to give here). But I'm just curious if there is an inherent performance impact of or-ing Q objects in a django query

your talking about oring together 80 conditions thats gonna be slow no matter what ... I would recommend figuring out a bettwe way to query for the records you want — Joran Beasley, Jan 05 '15 at 22:53
@doniyor No the model has few fields but I'm trying to find specific pairs of them (i.e. q1 = Q(field1=val1, field2=val2), q2 = Q(field1=val3, field2=val4), etc...) — sedavidw, Jan 05 '15 at 23:01
Invoke a Django shell and try out your code. My guess is the actual call to the database is really slow, and the rest of the code not so much. In any case, if you suffer performance issues: profile your code, see which line is slow. warath-coder is probably right about the index, but be sure to maintain the same order throughout your code: if you create an index on `(field1, field2)`, you should query on `Q(field1=..., field2=...)`, not `Q(field2=..., field1=...)`, or the composite index won't work. — knbk, Jan 06 '15 at 00:16
Have you tried the Debug Toolbar to see the SQL you're generating and to see how long it actually takes? Now 80 conditions is insane ^^, can you tell us (no code) what you're trying to achieve. There must be a better/simpler way. — François Constant, Jan 06 '15 at 01:48
I appreciate the suggestions, I wrote up AN answer that helped a lot, not sure if it's the best one — sedavidw, Jan 06 '15 at 14:14

sedavidw · Accepted Answer · 2021-03-02T02:19:11.523

Was able to shrink down the length of the query significantly by reducing the number of of Qobjects. They were all of the format like:

q1 = Q(field1=field1_val1, field2=field2_val1)
q2 = Q(field1=field1_val2, field2=field2_val2)
#...etc

I ended up grouping them by field1 values:

q_dict = {field1_val1: [all field_2 values paired with field1_val1], ...}

Then my q objects looked like this:

for field1_val, field2_vals = q_dict.items():
    query_params.append(Q(field1=field1_val, field2__in=field2_vals))

This ultimately shrank down my number of Q objects significantly and the query ran much faster

Performance impact of or-ing Q objects in django query

1 Answers1