Searching for numbers/product codes in Solr

Question

I have a text-field to which i'm feeding normal text that contains product codes (e.g. FI302010, RR220011, etc). I would like to be able to search for these product numbers. I have tried it out in the ANALYSIS-tool - And here it looks like it should be possible. A standard text-field does some word delimiting that splits the product codes into "FI" and "302010" - but as long as the same happens in both query and index it shouldn't be a problem. When i look at the schema browser i cant find the product codes, but i can find something that looks like a stemmed version of the code (i.e. 30201 - the last 0 is missing).

I have tried using different text-types (text_ws, textTight and text) - but none will allow me to do the search.

How should I enable text-search in text?

Thanks

Answer 1

Seems like you need to configure this field to use the solr.KeywordTokenizerFactory as its tokenizer.
As the name doesn't really convey, this tokenizer will handle the underlying field as one single token, leaving in place its with spaces, letter-to-number, number-to-letter transitions, punctuation and all (unless you also have char filters for these).

I'm not current on the latest specifics for for configuring Solr but I'm guessing the entry in the schema for the said field could look something like that:

<fieldtype name="myProductNumberFIeld" class="solr.TextField">
  <analyzer>
    <!-- can have a few filters of course -->
    <charFilter class="solr.MappingCharFilterFactory" 
       mapping="mapping-ISOLatin1Accent.txt"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <!-- etc. -->

    <!-- That's the main thing you need -->
    <tokenizer class="solr.KeywordTokenizerFactory"/>

  </analyzer>
</fieldtype>

More generally you'll want to define a rather specific analyzer for Product Number, Part Numbers, UPCs and such, as a way to improve recall on these fields as the users are not always aware of the specific format of these identifiers (where punctuation and other separators, if any, should be etc.).

The problem however of having a different analyzer is that it prevents associating the field with the general text search because at re trivial time, there can only be one way to process the user's query.

Answer 2

I found the solution here: http://search-with-fast-and-solr.blogspot.com/2010/12/maxfieldlength-in-solr.html

It turned out that solr was truncating the document at 10000 words - So I had to increase the maxFieldLength setting in solrconfig.xml

Thanks for your input!

Answer 3

This actually just works with the example configuration.

Taking stock Solr 1.4.1, and running the example config, you can see that, as you said, the analysis tool shows that this works for a stock text field.

Then, to verify, note that the "features" field in the example schema is of type text. Edit one of the sample files in apache-solr-1.4.1/example/exampledocs. Say "monitor.xml". Add the product code to the features field of one document. (and a space on either side)

Post the file to Solr (using java -jar post.jar monitor.xml). Now go search for it. It works. Check the schema browser. Show all the terms in the features field. Notice that the number is there by itself in a term.

It all works for me.

Not sure what happened to your schema browswer, perhaps a typo? Numbers aren't stemmed. It that trailing character is an o (letter) not a zero, that might be your problem.

Using keywordtokenizer will not allow you to find the numeric part of the product code, which was the original question.