6

Any ideas on how EdgeNgram treats numbers?

I'm running haystack with an ElasticSearch backend. I created an indexed field of type EdgeNgram. This field will contain a string that may contain words as well as numbers.

When I run a search against this field using a partial word, it works how it's supposed to. But if I put in a partial number, I'm not getting the result that I want.

Example:

I search for the indexed field "EdgeNgram 12323" by typing "edgen" and I'll get the index returned to me. If I search for that same index by typing "123" I get nothing.

Thoughts?

ComoWhat
  • 105
  • 1
  • 4

2 Answers2

4

I found my way here trying to solve this same problem in Haystack + Elasticsearch. Following the hints from uboness and ComoWhat, I wrote an alternate Haystack engine that (I believe) makes EdgeNGram fields treat numeric strings like words. Others may benefit, so I thought I'd share it.

from haystack.backends.elasticsearch_backend import ElasticsearchSearchEngine, ElasticsearchSearchBackend

class CustomElasticsearchBackend(ElasticsearchSearchBackend):
    """
    The default ElasticsearchSearchBackend settings don't tokenize strings of digits the same way as words, so emplids
    get lost: the lowercase tokenizer is the culprit. Switching to the standard tokenizer and doing the case-
    insensitivity in the filter seems to do the job.
    """
    def __init__(self, connection_alias, **connection_options):
        # see http://stackoverflow.com/questions/13636419/elasticsearch-edgengrams-and-numbers
        self.DEFAULT_SETTINGS['settings']['analysis']['analyzer']['edgengram_analyzer']['tokenizer'] = 'standard'
        self.DEFAULT_SETTINGS['settings']['analysis']['analyzer']['edgengram_analyzer']['filter'].append('lowercase')
        super(CustomElasticsearchBackend, self).__init__(connection_alias, **connection_options)

class CustomElasticsearchSearchEngine(ElasticsearchSearchEngine):
    backend = CustomElasticsearchBackend
yekta
  • 3,363
  • 3
  • 35
  • 50
Greg Baker
  • 101
  • 1
  • 6
3

if you're using the edgeNGram tokenizer, then it will treat "EdgeNGram 12323" as a single token and then apply the edgeNGram'ing process on it. For example, if min_grams=1 max_grams=4, you'll get the following tokens indexed: ["E", "Ed", "Edg", "Edge"]. So I guess this is not what you're really looking for - consider using the edgeNGram token filter instead:

If you're using the edgeNGram token filter, make sure you're using a tokenizer that actually tokenizes the text "EdgeNGram 12323" to produce two tokens out of it: ["EdgeNGram", "12323"] (standard or whitespace tokenizer will do the trick). Then apply the edgeNGram filter next to it.

In general, edgeNGram will take "12323" and produce tokens such as "1", "12", "123", etc...

uboness
  • 101
  • 1
  • I'm using the EdgeNGram analyser that comes with the elasticsearch backend in haystack. That uses a lowercase tokenizer and an EdgeNgram filter. If the field is "EdgeNgram 12323 myfield" and my search was for myfi, i will get the proper result. It just doesn't seem to treat numbers the same way. – ComoWhat Nov 30 '12 at 15:34
  • 4
    Nevermind, I figured it out. The lowercase tokenizer uses a letter tokenizer with a lowercase filter. This letter tokenizer will not tokenize numbers grouped together as one number but as separate numbers (ex: 1453 will be tokenized as 1, 4, 5, 3 instead of 1453). I changed it to use a standard tokenizer and just added the lowercase filter to my list of filters and it worked! – ComoWhat Nov 30 '12 at 22:10