Django Haystack Distinct Value for Field

Question

I am building a small search engine using Django Haystack + Elasticsearch + Django REST Framework, and I'm trying to figure out reproduce the behavior of a Django QuerySet's distinct method.

My index looks something like this:

class ItemIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    item_id = indexes.IntegerField(faceted=True)

    def prepare_item_id(self, obj):
        return obj.item_id

What I'd like to be able to do is the following:

sqs = SearchQuerySet().filter(content=my_search_query).distinct('item_id')

However, Haystack's SearchQuerySet doesn't have a distinct method, so I'm kind of lost. I tried faceting the field, and then querying Django using the returned list of item_id's, but this loses the performance of Elasticsearch, and also makes it impossible to use Elasticsearch's sorting features.

Any thoughts?

EDIT:

Example data:

Item Model
==========

id  title
1   'Item 1'
2   'Item 2'
3   'Item 3'


VendorItem Model << the table in question
================

id  item_id  vendor_id  lat   lon
1   1        1          38    -122
2   2        1          38.2  -121.8
3   3        2          37.9  -121.9
4   1        2          ...   ...
5   2        2          ...   ...
6   2        3          ...   ...

As you can see, there are multiple VendorItem's for the same Item, however when searching I only want to retrieve at most one result for each item. Therefore I need the item_id column to be unique/distinct.

I have tried faceting on the item_id column, and then executing the following query:

facets = SearchQuerySet().filter(content=query).facet('item_id')
counts = sqs.facet_counts()

# ids will look like: [345, 892, 123, 34,...]
ids = [i[0] for i in counts['fields']['item_id']]

items = VendorItem.objects.filter(vendor__lat__gte=latMin,
    vendor__lon__gte=lonMin, vendor__lat__lte=latMax,
    vendor__lon__lte=lonMax, item_id__in=ids).distinct(
        'item').select_related('vendor', 'item')

The main problem here is that results are limited to 100 items, and they cannot be sorted with haystack.

Not sure the approach is right. What are you trying to accomplish and what does the data look like — James R, Nov 18 '13 at 04:29
@JamesR I've added some information to the edit, would you mind taking a look? — Jeremy Blalock, Nov 18 '13 at 04:53

score 4 · Accepted Answer · answered Nov 18 '13 at 16:35

I think the best advice I can give you is to stop using Haystack.

Haystack's default backend (the elasticsearch_backend.py) is mostly written with Solr in mind. There are a lot of annoyances that I find in haystack, but the biggest has to be that it packs all queries into something called query_string. Using query string, they can use the lucene syntax, but it also means losing the entire elasticsearch DSL. The lucene syntax has some advantages, especially if this is what you are used to, but it is very limiting from an elasticsearch point of view.

Furthermore, I think you are applying an RDBMS concept to a search engine. That isn't to say that you shouldn't get the results you need, but the approach is often different.

The way you might query and retrieve this data might be different if you don't use haystack because haystack creates indexes in a way more appropriate for solr than for elasticsearch.

For example, in creating a new index, haystack will assign a "type" called "modelresult" to all models that will go in an index.

So, let's say you have some entities called Items and some other entities called vendoritems.

It might be appropriate to have them both in the same index but with vendoritems as a type of vendoritems and items having a type of items.

When querying, you would then query based on the rest endpoint so, something like localhost:9200/index/type (query). The way haystack achieves is this is through the django content types module. Accordingly, there is a field called "django_ct" that haystack queries and attaches to any query you might make when you are only looking for unique items.

To illustrate the above:

This endpoint searches accross all indexes

`localhost:9200/`

This endpoint searches across all types in an index:

`localhost:9200/yourindex/`

This endpoint searches in a type within an index:

`localhost:9200/yourindex/yourtype/`

and this endpoint searches in two specified types within an index:

`localhost:9200/yourindex/yourtype,yourothertype/`

Back to haystack though, you can possibly get unique values by adding a django_ct to your query, but likely that isn't what you want.

What you really want to do is a facet, and probably you want to use term facets. This could be a problem in haystack because it A.) analyzes all text and B.) applies store=True to all fields (really not something you want to do in elasticsearch, but something you often want to do in solr).

You can order facet results in elasticsearch (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html#_ordering)

I don't mean for this to be a slam on haystack. I think it does a lot of things right conceptually. It's especially good if all you need to do is index a single model (like say a blog) and just have it quickly return results.

That said, I highly recommend to use elasticutils. Some of the concepts from haystack are similar, but it uses the search dsl, rather than query_string (but you can still use query_string if you wanted).

Be warned though, I don't think you can order facets using elasticutils by default, but you can just pass in a python dictionary of the facets you want to facet_raw method (something I don't think you can do in haystack).

Your last option is to create your own haystack backend, inherit from the existing backend and just add some functionality to the .facet() method to allow for ordering per the above dsl.

Django Haystack Distinct Value for Field

1 Answers1

Linked