0

I have written a MapReduce program in which I am storing some part of output data into Hive table.

I have used Hive-JDBC driver to access Hive table via MapReduce code.

This program has compiled successfully on local machine.

After this, I created a JAR file and uploaded it on S3. Then I created an elasticmapreduce cluster and started it.

However, it is resulting into below mentioned errors:

java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201407161054_0001_m_000001_0: java.lang.ClassNotFoundException: org.apache.hadoop.hive.jdbc.HiveDriver attempt_201407161054_0001_m_000001_0: at java.net.URLClassLoader$1.run(URLClassLoader.java:366) attempt_201407161054_0001_m_000001_0: at java.net.URLClassLoader$1.run(URLClassLoader.java:355) attempt_201407161054_0001_m_000001_0: at java.security.AccessController.doPrivileged(Native Method) attempt_201407161054_0001_m_000001_0: at java.net.URLClassLoader.findClass(URLClassLoader.java:354) attempt_201407161054_0001_m_000001_0: at java.lang.ClassLoader.loadClass(ClassLoader.java:424) attempt_201407161054_0001_m_000001_0: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) attempt_201407161054_0001_m_000001_0: at java.lang.ClassLoader.loadClass(ClassLoader.java:357) attempt_201407161054_0001_m_000001_0: at java.lang.Class.forName0(Native Method) attempt_201407161054_0001_m_000001_0: at java.lang.Class.forName(Class.java:190) attempt_201407161054_0001_m_000001_0: at HubAndAuthority.InputHubMapper.configure(InputHubMapper.java:38) attempt_201407161054_0001_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) attempt_201407161054_0001_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) attempt_201407161054_0001_m_000001_0: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) attempt_201407161054_0001_m_000001_0: at java.lang.reflect.Method.invoke(Method.java:606)

It appears to be an issue of missing Hive-JDBC driver and it should get resolved by adding Hive-JDBC driver in classpath. However, I am not aware of the exact step to do this on Amazon's EMR.

Could you please let me know what is missing from my end and how to resolve it?

Thanks and Regards, Prafulla

jarlh
  • 42,561
  • 8
  • 45
  • 63
user3523860
  • 25
  • 4

1 Answers1

0

I'm not sure enough, but you should try this:

"Note If you want your custom classpath to override the original class path, you should set the environment variable, HADOOP_USER_CLASSPATH_FIRST to true so that the HADOOP_CLASSPATH value specified in hadoop-user-env.sh is first."

  1. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-config.html
  2. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-config_hadoop-user-env.sh.html

Regards, revet

revet
  • 1