How to remove duplicates from Multivalued Fields in SOLR?

Question

I tried the solutions listed in the below question.

Removing Solr duplicate values into multivalued field

I'm using dataimport handler and creating multiple values for the field using RegexTransformer.

My sql returns this for column FOO

Johnny Cash, Bonnie Money, Honey Bunny, Johnny Cash

and I store it to the multivalued field foo using splitBy=","

<field column="FOO" name="foo" splitBy=","/>

and it's stored in the multivalued field as such

{"Johnny Cash", "Bonnie Money", "Honey Bunny", "Johnny Cash"}

I've added this to the solrconfig xml

  <updateRequestProcessorChain name="distinctMultiValued" default="true">
    <!-- To remove duplicate values in a multivalued field-->
    <processor class="DistributedUpdateProcessorFactory"/>
    <processor class="UniqFieldsUpdateProcessorFactory">
        <str name="fieldRegex">foo</str>
    </processor>
    <processor class="solr.LogUpdateProcessorFactory" />        
    <processor class="solr.RunUpdateProcessorFactory" />
  </updateRequestProcessorChain>

I've also tried fieldName in place of fieldRegex and tried *oo instead of foo, but the duplicates remain.

Does this have something to do with the RegexTransformer?

I also have an update chain with TrimFieldsUpdateProcessorFactory than runs without any issues.

Razen · Accepted Answer · 2016-05-03T20:53:49.310

I was able to resolve this by moving the UniqFieldsUpdateProcessorFactory to the existing <updateRequestProcessorChain> block I had.

  <updateRequestProcessorChain name="skip-empty" default="true">
    <!--  Next two processors affect all fields - default configuration -->
    <processor class="TrimFieldUpdateProcessorFactory" />
    <processor class="RemoveBlankFieldUpdateProcessorFactory" />
    <processor class="UniqFieldsUpdateProcessorFactory">
        <str name="fieldRegex">.*oo</str>
    </processor>
    <processor class="solr.LogUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
  </updateRequestProcessorChain>

SOLR documentation UpdateRequestProcessorChain

At most one processor chain may be configured as the "default". if no processor is configured as a default, then an implicit default using LogUpdateProcessorFactory and RunUpdateProcessorFactory is created for you. Supplying a default processor chain may be the only way to affect documents indexed from some sources like the dataimport handler.

score 0 · Answer 2 · answered May 03 '16 at 14:13

0

may be you need to mention full class name in process class and <lst name="fields"> <str>multivaluedfield</str></lst>. like

<processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory"> <lst name="fields"> <str>multivaluedFieldXYZ</str> </lst> </processor>

answered May 03 '16 at 14:13

Vinod

1,965
1
9
18

That won't work. The fields option was deprecated as of SOLR 5. – Razen May 03 '16 at 14:41
In schema file add this filter to your multivaled field type definition `` – Vinod May 03 '16 at 15:03
Doesn't it just remove duplicate tokens. It doesn't remove duplicate values from the field in the query results, does it?. – Razen May 03 '16 at 15:51

score 0 · Answer 3 · answered Jun 28 '18 at 10:12

0

I am using solrJ to bind documents, and to avoid duplicated values I defined my multivalued field as a HashSet.

@Field("description") public Collection description = new HashSet<>();

answered Jun 28 '18 at 10:12

lgonzales

101
4

How to remove duplicates from Multivalued Fields in SOLR?

3 Answers3