0

I'm aware that this question may be closed, but I'm really having trouble with deciding what parts of a search feature should be implemented in jQuery, AJAX requests, or basic framework design.

In general the three tasks I'm looking at are executing new search queries, applying filters/sorting, and pagination. Currently all of these features are handled by Django (which is the framework I'm using) but I've found that this approach is both slow and inflexible. I realize that there is some part of this I should really move the javascript side of the site, but I'm having trouble deciding where to draw the distinction and I'm wondering if there are a set of Web Standards to help guide me.

Executing Search queries: Basically all I do here is collect the values in a few forms, then POST a JSON object to a REST api, getting the results back as JSON.

Filters/Sorting: I could do the filtering either by making a new AJAX call to the same REST api at a slightly different endpoint, or by filtering the old list of results based on data in the JSON object. For sorting, I currently have a python module that sorts the data in a number of different ways, but this requires a new get request for each different sorting, and the logic is simple enough that it would be easy to move to Javascript

Pagination: Fairly standard here with one change, which is that I could potentially implement pagination by making another AJAX call, as the REST api I'm using lets me define the exact bound of answers that I want to return, though if I want to use a non-standard sort it wouldn't really work. Right now I'm using Django's Pagination which works, but I feel like it would be much lighter on the app if I used a jQuery solution to pagination though.

I realize the responses will be partially opinion-based, but I'm really looking for concrete reasons why I should choose some of these options over others, such as security problems, performance, or significant deviation from Web Standards.

Slater Victoroff
  • 21,376
  • 21
  • 85
  • 144
  • 1
    I don't think there are any web standard, just common sense. For instance, searching/filtering in Javascript means the all the data has to be downloaded to the client. If it's large, that could be impractical. – Barmar Jul 30 '13 at 14:03
  • @Barmar the results aren't large enough to be impractical (currently capped at 100 results), but it seems like my code wouldn't be very transparent if I chose different methods for displaying search results based on how many there were. If I had a limited search and an unlimited search would it make sense to just put the limited version in javascript and have the unlimited version all work on the server side? That seems better performance-wise, but I don't think that code would be comprehensible to me. – Slater Victoroff Jul 30 '13 at 14:06
  • It doesn't matter if the results are capped at 100, as soon as you change the ordering or filtering, you'll be changing the window into your dataset and will be forced to load new data from the server. – Thomas Jul 30 '13 at 14:31

1 Answers1

2

On Django slowness

Currently all of these features are handled by Django (which is the framework I'm using) but I've found that this approach is both slow and inflexible.

Do you mean that you're executing you searches through Django's ORM? If so there are 2 reasons why this could be slow.

  1. You're querying on fields that are not indexed.
  2. You're attempting to do fulltext search ( or use __icontains ) which is slow in any database.

Django's ORM is a shim around the database. It has the same performance and flexibility limitation as the database you are using, just with a nicer API. If you want search performance and flexibility, use a datastore optimised for search, which leads me to...

Executing Search queries

Have you looked into Haystack? It offers an API custom tailored for performing searches on your data, and integrates with backends that are optimised for search speed and flexibility (my favourite is elasticsearch).

If you don't want to add extra tech to your stack, and are using postgres, why not look into GIN and GIST indexes for full-text-search. You will need to use some extensions to manage it though, and you will have to learn to profile your queries in order to add extra indexes where you need them.

Filters/Sorting

Always do this on the server. Unless you want to load an entire database's worth of query resuts into the browser in order to filter/sort, you have to do it on the server where you can be smart about discarding unneeded data early.

Pagination

Django's built-in pagination system can be a little lacking, but Haystack provides it's own pagination, and there's always the easy-to-use and mature django-paginator.

Notes

JQuery is a really nice way to make your search page spiffier, but moving everything into the browser is not going to be the answer to your problem. Use the tools at your disposal on the server side to make your API faster/more flexible, and then use that extra flexibility to make the client side even better.

Thomas
  • 11,757
  • 4
  • 41
  • 57
  • Already using elasticsearch, otherwise thank you very much for the advice! – Slater Victoroff Jul 30 '13 at 14:39
  • Also I feel like the filter/sorting point holds for sorting, but I think from a UX perspective it doesn't make sense to load anything new for a filter and I feel like it could make sense to just iterate through the already present results and hide results that should be filtered out. – Slater Victoroff Jul 30 '13 at 14:42
  • Also already using the django-paginator (as linked in the question), but it seems silly to me to send another request just to view a different part of the already-returned results. Additionally, using server-side pagination precludes adding any logic that would either add or remove entries on the front-end, which strikes me as a poor decision. – Slater Victoroff Jul 30 '13 at 14:44
  • Imagine that you have results loaded up to a limit, a small number of which have a certain property, then you apply a filter for that property. If you make no request to the server, you only have 4 or 5 results. If you make a request to the server, you could potentially have 100s of results. – Thomas Jul 30 '13 at 14:44
  • Yes, but I feel like that experience is very strange to a user. To have items that aren't actually in the original set of results, but will only show up once you scroll down after a filter. I think the case with having few results after a filter is much more congruent with user expectation. – Slater Victoroff Jul 30 '13 at 14:45
  • What do you mean to see another part of the already returned results? if you already have them, by all means display them. but if you run out of cached results, you'll obviously have to load more. – Thomas Jul 30 '13 at 14:48
  • the same logic with filters can be applied to changes in sorting. if you change the sorting method, you'll be putting items that are likely not in the cache to the top of the list – Thomas Jul 30 '13 at 14:49
  • My thought was just to have them all cached, I don't plan on having very many. – Slater Victoroff Jul 30 '13 at 14:56
  • So... sacrificing initial page load time to save on search time? – Thomas Jul 31 '13 at 02:03