SOLR External file field performance issue

Question

I am using SOLR 4.5(standalone instance) and I am trying to use external field to improve the ranking of documents. I have two external file fields for two different parameters which change daily which I use in "bf" and "boost" params of the edismax parser. Previously, these fields were part of the SOLR index.

I am facing serious performance issue for moving these fields out from index to external file. The CPU usage of SOLR machine reaches 100% in peak load and average response time has risen from 13 milliseconds to almost 150 milliseconds.

Is there anything I can do to improve the performance of SOLR when using external file fields. Are there any things to take care of while using external file field values within boost/bf functions ?

Would you share the `fieldType` and `field` of that external file from your schema.xml? — cheffe, Apr 07 '15 at 08:01
OK, that is the `fieldType`, how about the field where you actually use that type? — cheffe, Apr 08 '15 at 09:20

score 0 · Answer 1 · edited May 23 '17 at 10:26

0

As described in the SO Relevancy boosting very slow in Solr the key=value pairs the external file consists of, should be sorted by that key. This is also stated in the java doc of the ExternalFileField

The external file may be sorted or unsorted by the key field, but it will be substantially slower (untested) if it isn't sorted.

So if the content of your file would look like this (just an example)

300=3.8294805903e-07
5=3.8294805903e-07
20=3.8294805903e-07

You will need a script that alters the contents to

5=3.8294805903e-07
20=3.8294805903e-07
300=3.8294805903e-07

edited May 23 '17 at 10:26

Community

1
1

answered Apr 09 '15 at 11:15

cheffe

9,345
2
46
57

what if the key is alphanumeric string ? Should we sort that as well ? – user3257595 Apr 10 '15 at 11:40

SOLR External file field performance issue

1 Answers1