2

To achieve some degree of fault tolerance with Solr I have started to use the NGramFilterFactory. Here are the intersting bits from the schema.xml:

<field name="text" type="text" indexed="true" stored="true"/>
<copyField source="text" dest="text_ngram" />
<field name="text_ngram" type="text_ngram" indexed="true" stored="false"/>

<fieldType name="text_ngram" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
        <tokenizer class="solr.KeywordTokenizerFactory" />
        <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="3" />
    </analyzer>
</fieldType>

I am using the EDisMax query handler with pretty much the stock configuration. Here are the interesting lines from the solrconfig.xml:

  <requestHandler name="/browse" class="solr.SearchHandler">
    <lst name="defaults">
      <!-- Query settings -->
      <str name="defType">edismax</str>
      <str name="qf">
        name name_ngram^0.001
      </str>
      <str name="mm">100%</str>
      <str name="q.op">AND</str>
      ...

This works fine however gives me lots of irrelevant results. Using Solr's analyze capabilities I think I've tracked down the issue to the following cause:

The query is broken down into NGrams. Then Solr searches for either the tokenized query in the text field or one of the NGrams in the text_ngram field. Using debug=query will print out the following parsedquery when searching for "something":

(+DisjunctionMaxQuery(((text_ngram:som text_ngram:ome text_ngram:met text_ngram:eth text_ngram:thi text_ngram:hin text_ngram:ing) | text:something)))/no_coord

If I read this right it means that either

  1. One of the NGrams needs to match or
  2. The original query (tokenized) needs to match

Now this will also find items like "ethernet" as one of the NGrams (eth) is the same.

My question is: How can I set a higher threshold for the NGram matches? Is there a way to say "only return the item if at least 90% of the NGrams from the query match"? Making sure that 100% of the NGrams match would not make sense as this would effectively kill the fault tolerance.

Another way I thought of was to return only results that are above a certain score threshold relative to the top result. This is because the item "something" will have a very high relevancy compared to "ethernet". So is there a way to hook into Solr to return only results that have eg. at least 1/100th of the score of the top result? I read that there is a way to provide a custom HitCollector but I couldn't really find any info on this.

Thanks!

Georg M. Sorst
  • 264
  • 4
  • 13
  • why are you using ngrams during query time ? whats the use case to matching parts of search terms? Usually ngrams is used for prefix matching/faster wildcard matching. – Jayendra Jul 01 '13 at 11:35
  • There is [min match parameter](http://wiki.apache.org/solr/ExtendedDisMax#mm_.28Minimum_.27Should.27_Match.29) which you left as is. Please read the warning: yellow {!} – Jesvin Jose Jul 01 '13 at 11:36
  • @Jayendra: The idea was to achieve some kind of fault tolerant search. When someone searches for "someting" it should find "something". I figures this should be possible with NGrams. I know there is fuzzy search (~) but I didn't really find a way to implement this with the EDisMax query handler as I would have to append the ~ to every word. Any pointers? – Georg M. Sorst Jul 01 '13 at 15:18
  • You can check for phonetic filters as well which will be able to match misspell which sound similar – Jayendra Jul 01 '13 at 15:37
  • 1
    Any update here on how you fixed this? I face the same issue. – Krunal Oct 15 '13 at 12:57
  • Unfortunately not, sorry. We opted for not using NGrams. – Georg M. Sorst Oct 20 '13 at 17:11

1 Answers1

0

The idea was to achieve some kind of fault tolerant search. When someone searches for "someting" it should find "something"

Solr's SpellChecker does fuzzy search and you can set thresholds on it http://wiki.apache.org/solr/SpellCheckComponent .

ilinca
  • 1,343
  • 15
  • 17