Django optimize code for large data filtering

Question

As I am searching through 15000 results, is there any way to optimize the processing speed?

In my VIEW I'm filtering search as this:

if form.is_valid():

      results = Screening.objects.filter(
         screening_date__range(form.cleaned_data['start_date'],form.cleaned_data['end_date']))

      if form.cleaned_data['company']:
          results = results.filter(company=form.cleaned_data['company'])
      if form.cleaned_data['company_job']:
          results = results.filter(company_job__in=form.cleaned_data['company_job'])
      if form.cleaned_data['company_job_type']:
          results = results.filter(company_job_type=form.cleaned_data['company_job_type'])
      if form.cleaned_data['reason']:
          results = results.filter(reason_for_testing__in=form.cleaned_data['reason'])`

And in TEMPLATE, the passed result is used as:

{% for result in results %}

      <td>{{ result.company.name}}</td>
      <td>{{ result.first_name }} {{ result.last_name }}</td>
      <td>{{ result.company_job.job_number }}</td>
      <td>{{ result.id }}</td>
      <td>{{ result.screening_date|date}}</td></tr>

Is there any way to optimize the processing or should I use cache or sth in this case?

What exactly is your problem? Is the resulting query too slow? Do you have indices set correctly? — George Karpenkov, Jan 24 '12 at 10:58
Database only gets hit when query is evaluated, so is there is anything i can do to optimize my code in view. I think there is also problem with indices. — Abhaya, Jan 24 '12 at 11:24

score 0 · Answer 1 · answered Jan 24 '12 at 11:38

0

You should definitely use select_related to make ony one DB hit (not N).

Then profile the code, find the bottleneck and try to optimize it, not "optimize in general".

answered Jan 24 '12 at 11:38

DrTyrsa

31,014
7
86
86

score 0 · Accepted Answer · answered Jan 24 '12 at 16:09

This is not an answer, but just a tip to make your code easier to read and work with:

  filters = {
      'screening_date__range': (form.cleaned_data['start_date'],form.cleaned_data['end_date'])
  }

  if form.cleaned_data['company']:
      filters['company'] = form.cleaned_data['company']
  if form.cleaned_data['company_job']:
      filters['company_job__in'] = form.cleaned_data['company_job']
  if form.cleaned_data['company_job_type']:
      filters['company_job_type'] = form.cleaned_data['company_job_type']
  if form.cleaned_data['reason']:
      filters['reason_for_testing__in'] = form.cleaned_data['reason']

  Screening.objects.filter(**filters)

Django optimize code for large data filtering

2 Answers2