sqoop2 not finding log4j2 from hadoop

Question

I am trying to install sqoop2 (1.99.7) on my ubuntu server. I am trying to follow the instructions provided on the apache website here. I have a working hadoop installation and I have downloaded and extracted the sqoop file to the /usr/local/sqoop location.

tar -xvf sqoop-1.99.7-bin-hadoop200.tar.gz
mv sqoop-1.99.7-bin-hadoop200 /usr/local/sqoop

I believe I have all the environmental variables defined, in particular HADOOP_HOME which I thought is stated to direct where sqoop looks for the jar files.

However, when I try to verify installation with sqoop2-tool verify I get the following output.

Setting conf dir: /usr/local/sqoop/bin/../conf
Sqoop home directory: /usr/local/sqoop
Sqoop tool executor:
        Version: 1.99.7
        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
        at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:36)
        at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
        at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
        at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
        at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 5 more

Somehow, it is failing to find the log4j2 configuration file. I'm not sure why this is the case.

This question is similar to the one here but the solution provided does not help. If I modify the sqoop.properties file and point directly to the hadoop configuration directory /usr/local/hadoop/etc/hadoop (which is where my core-site.xml, hdfs-site.xml, etc. are located) I continue to get the error above.

EDIT

Output of grep -r "org.apache.hadoop.conf.Configuration" /usr/local/hadoop | grep jar

Binary file /usr/local/hadoop/share/hadoop/common/sources/hadoop-common-2.8.0-sources.jar matches
Binary file /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.0.jar matches
Binary file /usr/local/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/hadoop-common-2.8.0.jar matches
Binary file /usr/local/hadoop/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/hadoop-common-2.8.0.jar matches

Answer 1

Sqoop.properties is a Java property file. Environment variable should be defined in sqoop-env.sh or set it up using export command.

Can you try to execute the below environment variables export command before executing the sqoop command, It it works you can add these commands to sqoop-env.sh environment file.

export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_YARN_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop

Make sure /usr/local/hadoop is correct.

Edit - If you look at the last line of sqoop command, it's a bash script and it uses hadoop command internally to invoke sqoop class, so all hadoop related libs will be loaded to sqoop environment, if HADOOP_COMMON_HOME env variable is correct.

Are you able to execute hadoop commands in this server ?, Can you share the output of ${HADOOP_COMMON_HOME}/bin/hadoop fs -ls / ; If this works, this error could be due to compatibility - Sqoop version may not compatible with Hadoop.