Multi word synonyms in solr

Question

I'm trying to implement multi-word synonyms in solr, specifically of the type

msc divina => divina

So, if a user enters "msc divina", solr should return results for "divina" only.

The definition in schema.xml looks like this:

<fieldType name="text_de" class="solr.TextField" positionIncrementGap="100" 
    autoGeneratePhraseQueries="true">
    <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.SynonymFilterFactory"
            synonyms="synonyms_de.txt"
            ignoreCase="true"
            expand="false" />
        <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords_de.txt"
            enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory"
            generateWordParts="1"
            generateNumberParts="1"
            catenateWords="1"
            catenateNumbers="1"
            catenateAll="0"
            splitOnCaseChange="1" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.KeywordMarkerFilterFactory" 
            protected="protwords_de.txt" />
        <filter class="solr.SnowballPorterFilterFactory" language="German2" />
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords_de.txt"
            enablePositionIncrements="true" />
        <filter class="solr.WordDelimiterFilterFactory"
            generateWordParts="1"
            generateNumberParts="1"
            catenateWords="0"
            catenateNumbers="0"
            catenateAll="0"
            splitOnCaseChange="1" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.KeywordMarkerFilterFactory" 
            protected="protwords_de.txt" />
        <filter class="solr.SnowballPorterFilterFactory" language="German2" />
    </analyzer>
</fieldType>

It doesn't work. If I add a synonym filter to the query analyzer, a search on "msc divina" returns every hit for "msc and "divina".

How can I solve this?

Refer to this http://stackoverflow.com/a/43164795/3593530 stackoverflow answer. This gets fixed in solr-6.5.0 — Sanjay Dutt, Apr 02 '17 at 05:02

score 3 · Answer 1 · answered May 15 '18 at 12:56

Starting from Solr 6.4 for multi-word synonyms you need to use solr.SynonymGraphFilterFactory

This filter maps single- or multi-token synonyms, producing a fully correct graph output. This filter is a replacement for the Synonym Filter, which produces incorrect graphs for multi-token synonyms.

If you use this filter during indexing, you must follow it with a Flatten Graph Filter to squash tokens on top of one another like the Synonym Filter, because the indexer can’t directly consume a graph. To get fully correct positional queries when your synonym replacements are multiple tokens, you should instead apply synonyms using this filter at query time.

Example of the analyzer for index time:

<analyzer type="index">
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.SynonymGraphFilterFactory" synonyms="mysynonyms.txt"/>
  <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters -->
</analyzer>

Since now token streams are graphs - proper arcs would be provided for multiword synonyms for a file

fast → speedy
wi fi → wifi
wi fi network → hotspot

In this case - multiwords would work properly.

Reference to McCandless blog post - http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html

score 1 · Answer 2 · answered Nov 12 '13 at 15:01

From Solr documentation:

Keep in mind that while the SynonymFilter will happily work with synonyms containing multiple words (ie: "sea biscuit, sea biscit, seabiscuit") The recommended approach for dealing with synonyms like this, is to expand the synonym when indexing. This is because there are two potential issues that can arrise at query time:

The Lucene QueryParser tokenizes on white space before giving any text to the Analyzer, so if a person searches for the words sea biscit the analyzer will be given the words "sea" and "biscit" seperately, and will not know that they match a synonym. Phrase searching (ie: "sea biscit") will cause the QueryParser to pass the entire string to the analyzer, but if the SynonymFilter is configured to expand the synonyms, then when the QueryParser gets the resulting list of tokens back from the Analyzer, it will construct a MultiPhraseQuery that will not have the desired effect. This is because of the limited mechanism available for the Analyzer to indicate that two terms occupy the same position: there is no way to indicate that a "phrase" occupies the same position as a term. For our example the resulting MultiPhraseQuery would be "(sea | sea | seabiscuit) (biscuit | biscit)" which would not match the simple case of "seabiscuit" occuring in a document

In here they describe one problem: you can not search for sea biscit and get a match on indexed seabiscuit, unless you use expand=true, but they also explain what happens at the query time with a multi word query which is your case.

msc divina -> msc | divina - phrase query

which will match both msc and divina documents. If you can specify at query time that you a searching for "msc divina" it will work.

Otherwise you need either a multi-word aware tokenizer at the query time or you can expand the FieldQParser plugin to do this for you. You can find more here.

Thanks for the answer. This is where I get confused: "expand the synonym when indexing", which agrees with what you say above. But if expand=true (in the index analyzer), do the synonyms expand to "msc", "divina" and "msc divina", because I don't want that. I want "msc divina" to be a synonym for "divina". The other confusion is whether I should have a synonym filter on the query analyzer too, and what should the expand value there be? — midnig, Nov 13 '13 at 08:39
It looks like there is a patch submitted for https://issues.apache.org/jira/browse/SOLR-5379 that deals with the same problem. Check also the related jira issues for additional information. Good luck — Ion Cojocaru, Nov 13 '13 at 15:11

score 1 · Answer 3 · answered Jan 01 '18 at 08:07

Here is a solution you will find on internet: https://dzone.com/articles/solution-multi-term-synonyms

Other than that, my solution to this problem was domain specific. In my case, I was certain about my query lengths (i.e. less than 200 or there are only 5-10 words).

I have replaced spaces with underscores in synonym entries. Here is one of my synonym entries:
```
"like_to":["love_to","loves_to","need_to","needs_to"]
```
Use KeywordTokenizerFactory to send full query for filtering
```
<tokenizer class="solr.KeywordTokenizerFactory"/>
```
Use ShingleFilterFactory to index/query all possible sub-phrases of sizes between minShingleSize and mazShingleSize.
```
<filter class="solr.ShingleFilterFactory" minShingleSize="2" outputUnigrams="true" maxShingleSize="3"/>
```

Then use PatternReplaceCharFilterFactory to replace whitespaces to underscores (_)

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="\\s+" replacement="_"/>

Use your synonym filter factory.

Example

Query: I love to travel

Tokens: I love, I love to, love to, love to travel, to travel, tavel

Replaced with _: I_love, I_love_to, love_to, love_to_travel, to_travel, tavel

Synonym filter turns these into: I_love, I_love_to, like_to, love_to_travel, to_travel, tavel

So, it will eventually change the love to phrase to like to.

Hope this trick helps although it involves costly operations.

Multi word synonyms in solr

3 Answers3

Example