How to resolve NoClassDefFoundError while using Hadoop?

Question

I am getting

Exception in thread "main" java.lang.NoClassDefFoundError: com/linkedin/camus/etl/IEtlKey.

On running the command:

hadoop jar camus-etl-kafka-0.1.0-SNAPSHOT.jar 
com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

I am getting the below exceptions..

2016-04-27 11:34:04.622 java[13567:351959] Unable to load realm mapping info from SCDynamicStore
[NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: com/linkedin/camus/etl/IEtlKey
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:252)
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:235)
    at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:691)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.linkedin.camus.etl.kafka.CamusJob.main(CamusJob.java:646)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: com.linkedin.camus.etl.IEtlKey

I have included camus-example-0.1.0-SNAPSHOT-shaded.jar in the classpath .

Please let me know if I am missing something .

Thanks in Advance

Soumyajit

abarisone · Answer 1 · 2016-04-27T12:07:48.633

You should try to include camus-api you can find on this LinkedIn's previous generation Kafka to HDFS pipeline page, since the missing class is contained in this package, as you can see here.

Pay attention to other transitive dependencies that may be required by Camus.

In addition, to be sure that classes will be found in the classpath when you use the hadoop jar from command line, you can add the libjars command line option, as reported in Using the libjars option with Hadoop:

$ export LIBJARS=/path/jar1,/path/jar2
$ hadoop jar my-example.jar com.example.MyTool -libjars ${LIBJARS} -mytoolopt value

It could be useful to know that Camus is going to be superseded by Gobblin:

Camus is being phased out and replaced by Gobblin. For those using or interested in Camus, we suggest taking a look at Gobblin.

For instructions on Migrating from Camus to Gobblin, please take a look at Camus Gobblin Migration.

Hi.. Thanks for your suggestion . I have already added camus-api in the classpath , but still no luck ..Let me know if I am missing anything.. — Soumyajit Chatterjee, Apr 27 '16 at 09:33
yes.. Actually when I am running from IDE, it is not giving the exception. but when I am trying to run it from terminal, it is giving the same exception . — Soumyajit Chatterjee, Apr 27 '16 at 10:15

How to resolve NoClassDefFoundError while using Hadoop?

1 Answers1