Sqoop returns "response status of 404 Not Found"

Question

I have installed latest Hadoop 2.5.1 and Sqoop 1.99.3 (from tarball files) on CentOS 7. Sqoop was installed according the instructions from here: http://sqoop.apache.org/docs/1.99.3/Installation.html

I then wanted to test the Sqoop server according to this page: http://sqoop.apache.org/docs/1.99.3/Sqoop5MinutesDemo.html

I ran the following command:

show version --all

and got the following error message:

Exception: com.sun.jersey.api.client.UniformInterfaceException Message: GET http://localhost:12000/sqoop/version returned a response status of 404 Not Found

This is what the config and log files contain immediately after I tried to start the Sqoop server (by running Sqoop.sh server start):

Sqoop variables in /etc/bashrc:

export SQOOP_HOME="/usr/local/sqoop"
export SQOOP_CONF_DIR=$SQOOP_HOME/server/conf

/usr/local/sqoop/server/conf/Catalina.properties:

/usr/local/hadoop/share/hadoop/common/*.jar,
/usr/local/hadoop/share/hadoop/common/lib/*.jar,
/usr/local/hadoop/share/hadoop/hdfs/*.jar,
/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,
/usr/local/hadoop/share/hadoop/mapreduce/*.jar,
/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,
/usr/local/hadoop/share/hadoop/tools/*.jar,
/usr/local/hadoop/share/hadoop/tools/lib/*.jar,
/usr/local/hadoop/share/hadoop/yarn/*.jar,
/usr/local/hadoop/share/hadoop/yarn/lib/*.jar

/usr/local/sqoop/server/conf/sqoop.properties:

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop/

/usr/local/sqoop/server/logs/localhost.2014-11-13.log:

SEVERE: Exception sending context initialized event to listener instance of class org.apache.sqoop.server.ServerInitializer
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

/usr/local/sqoop/log/sqoop.log:

2014-11-13 08:19:55,466 INFO  audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initialize(AuditLoggerManager.java:79)] Begin audit logger manager initialization
2014-11-13 08:19:55,470 WARN  audit.FileAuditLogger [org.apache.sqoop.audit.FileAuditLogger.initialize(FileAuditLogger.java:67)] appender: log4j.appender.defaultAppender
2014-11-13 08:19:55,471 INFO  audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initializeLoggers(AuditLoggerManager.java:126)] Audit Logger has been initialized: default
2014-11-13 08:19:55,471 INFO  audit.AuditLoggerManager [org.apache.sqoop.audit.AuditLoggerManager.initialize(AuditLoggerManager.java:85)] Audit logger manager initialized: OK
2014-11-13 08:19:55,472 INFO  repository.RepositoryManager [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:80)] Setting system properties: {derby.stream.error.file=/usr/loc
2014-11-13 08:19:55,478 INFO  repository.JdbcRepositoryContext [org.apache.sqoop.repository.JdbcRepositoryContext.<init>(JdbcRepositoryContext.java:154)] [repo-ctx] handler=org.apache.sqoop.repository.derb
2014-11-13 08:19:55,683 INFO  derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.initialize(DerbyRepositoryHandler.java:207)] DerbyRepositoryHandler initialized.
2014-11-13 08:19:55,686 INFO  repository.JdbcRepositoryProvider [org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:160)] JdbcRepositoryProvider init
2014-11-13 08:19:55,686 INFO  repository.JdbcRepositoryProvider [org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:67)] JdbcRepository initialized.
2014-11-13 08:19:55,686 INFO  repository.RepositoryManager [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:118)] Creating or upgrading on disk structures if necessary
2014-11-13 08:19:56,022 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2014-11-13 08:19:56,022 INFO  repository.JdbcRepository [org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:130)] Creating repository schema objects
2014-11-13 08:19:56,198 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectVersion(DerbyRepositoryHandler.java:272)] Detecting old version of repository
2014-11-13 08:19:56,354 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-13 08:19:56,354 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-13 08:19:56,354 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-13 08:19:56,360 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2014-11-13 08:19:56,375 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectVersion(DerbyRepositoryHandler.java:272)] Detecting old version of repository
2014-11-13 08:19:56,375 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-13 08:19:56,376 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-13 08:19:56,376 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-13 08:19:56,376 INFO  repository.RepositoryManager [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:128)] Repository initialized: OK
2014-11-13 08:19:56,377 INFO  connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:174)] Connector config urls: [jar:file:/usr/local/sqoop-1.99.3-bin-had
2014-11-13 08:19:56,378 DEBUG connector.ConnectorHandler [org.apache.sqoop.connector.ConnectorHandler.<init>(ConnectorHandler.java:58)] Connector configuration: {org.apache.sqoop.connector.class=org.apache
2014-11-13 08:19:56,415 INFO  connector.ConnectorHandler [org.apache.sqoop.connector.ConnectorHandler.<init>(ConnectorHandler.java:113)] Connector [org.apache.sqoop.connector.jdbc.GenericJdbcConnector] ini
2014-11-13 08:19:56,416 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2014-11-13 08:19:56,416 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 2, rollback: false
2014-11-13 08:19:56,416 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:441)] Looking up connector: generic-jdbc-conne
2014-11-13 08:19:56,477 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:488)] Looking up connector: generic-jdbc-conne
2014-11-13 08:19:56,477 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 2, rollback: false
2014-11-13 08:19:56,477 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 1, rollback: false
2014-11-13 08:19:56,478 DEBUG connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:227)] Registered connector: connector-generic-jdbc-connector:1
2014-11-13 08:19:56,478 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-13 08:19:56,478 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-13 08:19:56,478 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-13 08:19:56,478 INFO  connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:200)] Connectors loaded: {generic-jdbc-connector={generic-jdbc-connect
2014-11-13 08:19:56,486 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2014-11-13 08:19:56,486 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findFramework(DerbyRepositoryHandler.java:537)] Looking up framework metadata
2014-11-13 08:19:56,503 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findFramework(DerbyRepositoryHandler.java:583)] Looking up framework metadta found: fram
2014-11-13 08:19:56,503 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-13 08:19:56,503 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-13 08:19:56,503 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-13 08:19:56,504 INFO  framework.FrameworkManager [org.apache.sqoop.framework.FrameworkManager.initialize(FrameworkManager.java:159)] Submission manager initialized: OK
2014-11-13 08:19:56,507 INFO  mapreduce.MapreduceSubmissionEngine [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:75)] Initializing Map-reduce Sub
2014-11-13 08:19:56,512 INFO  core.SqoopServer [org.apache.sqoop.core.SqoopServer.destroy(SqoopServer.java:35)] Shutting down Sqoop server

According to the last line in the log file, the Sqoop server shutdown itself and doesn't start. I googled every possible combination of error messages above and didn't find anything that could have helped. How can I troubleshoot this problem?

Answer 1

This is common when the hadoop libs can't be found. There are two things you can do:

1. Install Sqoop through apache Big Top or through one of the Hadoop distributors (ie Cloudera)
2. Edit the tomcat configuration to include these jars (as you've tried above). The log file is giving you specifics in terms of which classes are missing. You can search for your missing jars via the command line.

I highly recommend the first option.

Answer 2

Thanks! I double checked the catalana.properties file, and made sure the common.loader section is a single line. If this section consists of the multiple lines, the TomCat will not load Sqoop webapp. Line breaks are not accepted.