Cannot get DataImportHandler working for Solr in Tomcat

Question

I am trying to add a DataImportHandler to my Solr 4.10.2 installation on CentOS 6 in a Tomcat 6 container. The purpose is to index data from a MySQL database to make it searchable.

Therefore I created a Solr-home directory at /var/www/solr . Inside this folder I have the structure

solr.xml
zoo.cfg
conf/
    data-config.xml  
    schema.xml
    solrconfig.xml
dist/
    ...
    solr-dataimporthandler-4.10.2.jar
    solr-dataimporthandler-extras-4.10.2.jar
    ...
lib/
     mysql-connector-java-5.1.34-bin.jar

If I start Tomcat I can access the Solr frontpage via http://127.0.0.1:8088/solr/. But if I go to http://127.0.0.1:8088/solr/dataimport I get an error message

"HTTP Status 404 - /solr/dataimport"
type: Status report
message: /solr/dataimport
description: The requested resource (/solr/dataimport) is not available.

Why is it not possible for me to get the dataimport handler working? I couldn't find an answer in other threads.

My Configuration XML file contains the following:

solrconfig.xml

<requestHandler name="/dataimport"
        class="org.apache.solr.handler.dataimport.DataImportHandler">
        <lst name="defaults">
                <str name="config">/var/www/solr/conf/data-config.xml</str>
        </lst>
</requestHandler>

<lib dir="/var/www/solr/dist/" regex="solr-dataimporthandler-\d.*\.jar" />

EDIT:

The Tomcat Logs (var/log/tomcat6/catalina.out) for the startup of the Tomcat servlet are shown below

    INFO: Server startup in 3722 ms
Jan 23, 2015 9:46:08 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
Jan 23, 2015 9:46:09 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Jan 23, 2015 9:46:09 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Jan 23, 2015 9:46:09 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [org.apache.solr.schema.DateField.ThreadLocalDateFormat] (value [org.apache.solr.schema.DateField$ThreadLocalDateFormat@29215f06]) and a value of type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat] (value [org.apache.solr.schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
Jan 23, 2015 9:46:09 AM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
Jan 23, 2015 9:46:11 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jan 23, 2015 9:46:11 AM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jan 23, 2015 9:46:11 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 986 ms
Jan 23, 2015 9:46:11 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jan 23, 2015 9:46:11 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.24
Jan 23, 2015 9:46:11 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor manager.xml
Jan 23, 2015 9:46:12 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor host-manager.xml
Jan 23, 2015 9:46:12 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive solr.war
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/tomcat6/slf4j-log4j12-1.7.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/tomcat6/slf4j-jcl-1.7.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/tomcat6/slf4j-jdk14-1.7.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/tomcat6/slf4j-nop-1.7.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/tomcat6/slf4j-simple-1.7.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
0    [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  – SolrDispatchFilter.init()
44   [main] INFO  org.apache.solr.core.SolrResourceLoader  – Using JNDI solr.home: /var/www/solr
50   [main] INFO  org.apache.solr.core.SolrResourceLoader  – new SolrResourceLoader for directory: '/var/www/solr/'
66   [main] INFO  org.apache.solr.core.SolrResourceLoader  – Adding 'file:/var/www/solr/lib/mysql-connector-java-5.1.34/' to classloader
68   [main] INFO  org.apache.solr.core.SolrResourceLoader  – Adding 'file:/var/www/solr/lib/mysql-connector-java-5.1.34-bin.jar' to classloader
619  [main] INFO  org.apache.solr.core.ConfigSolr  – Loading container configuration from /var/www/solr/solr.xml
876  [main] INFO  org.apache.solr.core.CoresLocator  – Config-defined core root directory: /var/www/solr
896  [main] INFO  org.apache.solr.core.CoreContainer  – New CoreContainer 580871917
897  [main] INFO  org.apache.solr.core.CoreContainer  – Loading cores into CoreContainer [instanceDir=/var/www/solr/]
922  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting socketTimeout to: 0
923  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting urlScheme to: null
935  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting connTimeout to: 0
936  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting maxConnectionsPerHost to: 20
937  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting corePoolSize to: 0
939  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting maximumPoolSize to: 2147483647
939  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting maxThreadIdleTime to: 5
941  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting sizeOfQueue to: -1
942  [main] INFO  org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting fairnessPolicy to: false
1361 [main] INFO  org.apache.solr.update.UpdateShardHandler  – Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
1368 [main] INFO  org.apache.solr.logging.LogWatcher  – SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
1381 [main] INFO  org.apache.solr.logging.LogWatcher  – Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
1385 [main] INFO  org.apache.solr.core.CoreContainer  – Host Name: 
1511 [main] INFO  org.apache.solr.core.CoresLocator  – Looking for core definitions underneath /var/www/solr
1544 [main] INFO  org.apache.solr.core.CoresLocator  – Found 0 core definitions
1550 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  – user.dir=/usr/share/tomcat6
1552 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  – SolrDispatchFilter.init() done
Jan 23, 2015 9:46:15 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory sample
Jan 23, 2015 9:46:15 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory examples
Jan 23, 2015 9:46:16 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Jan 23, 2015 9:46:16 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jan 23, 2015 9:46:16 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jan 23, 2015 9:46:16 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/61  config=null
Jan 23, 2015 9:46:16 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4791 ms

Also there is a discussion about a similar issue [https://stackoverflow.com/questions/10343756/how-to-use-the-solr-data-import-handler-to-index-a-mysql-table] (https://stackoverflow.com/questions/10343756/how-to-use-the-solr-data-import-handler-to-index-a-mysql-table) — notdang, Jan 22 '15 at 18:33
@ notdang: I added the logs for the startup. I can't see an error in there.. I have seen the other post and answers as well but unfortunately they didn't help. — seyfe, Jan 22 '15 at 19:34
Looking at your config, it doesn't seem that all the libs are added — notdang, Jan 22 '15 at 20:01
possible duplicate of [org.apache.solr.common.SolrException: Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler](http://stackoverflow.com/questions/13913915/org-apache-solr-common-solrexception-error-loading-class-org-apache-solr-handl) — kenorb, Mar 29 '15 at 13:54

notdang · Accepted Answer · 2015-01-25T03:06:26.200

1

I don't think your regex is matching both DIH related jars. Try changing the config to:

<lib dir="/var/www/solr/dist/" regex="solr-dataimporthandler-.*\.jar" />

Or, as an alternative, you can try to specify the full paths to each lib:

<lib path="/var/www/solr/dist/solr-dataimporthandler-4.10.2.jar" />  
<lib path="/var/www/solr/dist/solr-dataimporthandler-extras-4.10.2.jar" />

UPDATE After some troubleshooting it seems that the problem was that Solr could not autodetect the cores, so the solrconfig.xml was not loaded at all. So the solution was to add the core.properties file to the core folder.

edited Jan 25 '15 at 03:06

answered Jan 22 '15 at 20:05

notdang

500
2
8

This wouldn't work either (none of the two possibilites).. is there a way to confirm that solr has looked at the paths and loaded the .jar files? – seyfe Jan 22 '15 at 20:47
I started a Solr instance and this is what I see in my logs: Jan 22, 2015 5:59:39 PM org.apache.solr.core.SolrResourceLoader replaceClassLoader INFO: Adding 'file:/tmp/solr/solr/dist/apache-solr-velocity-4.0.0.jar' to classloader – notdang Jan 23 '15 at 00:00
Is this also in your catalina.out or is there another log file I have to look at? – seyfe Jan 23 '15 at 09:38
I misconfigured my solr logging. If you look again at my entry post I have updated the logfile. It loads `66 [main] INFO org.apache.solr.core.SolrResourceLoader – Adding 'file:/var/www/solr/lib/mysql-connector-java-5.1.34/' to classloader` but there is no DIH file mentioned.. – seyfe Jan 23 '15 at 09:52
I saw people adding `` to solrconfig.xml but in the downloaded solr archive `/contrib/dataimporthandler/` is an empty folder except for a README file. Might this be the problem? However, the logfile should at least state loading the other dataimporthandler .jar file – seyfe Jan 23 '15 at 12:02
in my contrib/dataimporthandler I see these files: activation-1.1.jar mail-1.4.1.jar – notdang Jan 23 '15 at 13:49
I have activation-1.1.1.jar, gimap-1.5.1.jar, and javax.mail-1.5.1.jar in `contrib/dataimporter-extras/lib`. Adding this folder to my solrconfig.xml doesn't help though. Is it possible that my solrconfig.xml is not read? – seyfe Jan 23 '15 at 14:37
You could deliberately make a mistake there and see if solr would start. – notdang Jan 23 '15 at 18:17
adding some nonsense: ` – seyfe Jan 24 '15 at 12:10
You can view your solrconfig.xml through the UI – notdang Jan 24 '15 at 17:23
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/69512/discussion-between-notdang-and-seyfe). – notdang Jan 24 '15 at 17:25

Cannot get DataImportHandler working for Solr in Tomcat

1 Answers1