NoClassDefFoundError with gradle, giraph, and hadoop

Question

So, I've been looking around a lot and I haven't found a good answer to my question, and this is driving me crazy, so I figured I'd ask here and hopefully I can get help. I'm trying to do automated testing in a Giraph project using gradle. I'm a total beginner with gradle. Just to get started, I copied the test code for the SimpleShortestPathComputation class into my project, to make sure I could get the tests up and running. When I do gradle test, however, I get the following error:

$ gradle test --info
<skipping some output here...>

Successfully started process 'Gradle Test Executor 1'
Gradle Test Executor 1 started executing tests.
WCCTest > testToyData FAILED
    java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:142)
        at WCCTest.testToyData(WCCTest.java:180)

    Caused by:
    java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        ... 2 more

Gradle Test Executor 1 finished executing tests.
WCCTest > testOnShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at     org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnShorterPathFound(WCCTest.java:64)

WCCTest > testToyDataJson FAILED
    java.lang.NoClassDefFoundError: Could not initialize class org.apache.giraph.conf.GiraphConfiguration
        at WCCTest.testToyDataJson(WCCTest.java:127)

WCCTest > testOnNoShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnNoShorterPathFound(WCCTest.java:95)

4 tests completed, 4 failed
<more output...>
:test FAILED
:test (Thread[main,5,main]) completed. Took 1.663 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///...build/reports/tests/index.html

BUILD FAILED

I'm using the totally standard project directory structure, and this is my build.gradle file:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    compile files('$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$HADOOP_HOME/hadoop-core-0.20.203.0.jar')

    testCompile group: 'junit', name: 'junit', version: '4.+'
    testCompile group: 'org.mockito', name: 'mockito-all', version: '1.9.5'
}

It compiles with no problem, and the jar files I'm including as dependences include the classes for which it says NoClassDefFoundError (according to jar tf). Any ideas what I'm doing wrong? Thanks in advance.

Bill Lin · Answer 1 · 2014-07-08T06:48:05.907

0

try add following into your build.gradle

    println("HADOOP_HOME=$HADOOP_HOME")
    compile files("$HADOOP_HOME/hadoop-core-0.20.203.0.jar")
    println("System.env.HADOOP_HOME=$System.env.HADOOP_HOME")
    compile files("$System.env.HADOOP_HOME/hadoop-core-0.20.203.0.jar")

edited Jul 08 '14 at 06:48

answered Jul 07 '14 at 15:09

Bill Lin

1,145
1
11
12

Hey, thanks for the help. So actually: `$ jar tf hadoop-core-0.20.203.0.jar | grep Configur` outputs: `org/apache/hadoop/conf/Configurable.class org/apache/hadoop/conf/Configuration$1.class org/apache/hadoop/conf/Configuration$IntegerRanges$Range.class org/apache/hadoop/conf/Configuration$IntegerRanges.class org/apache/hadoop/conf/Configuration.class` Plus some other stuff. Also when I use that compile group I get some other errors: `java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration` `javax.security.auth.login.LoginException: java.lang.NoClassDefFoundError`etc – Matthew Saltz Jul 07 '14 at 21:57
Not sure of etiquette here, should I add the full new output to the original post? – Matthew Saltz Jul 07 '14 at 22:01
Also, could you elaborate on using $HADOOP_HOME? Do you mean putting that inside `compile files($HADOOP_HOME)` like that? If I do that it doesn't seem to include anything at all, and I have to include the core jar anyways. I really appreciate the help btw – Matthew Saltz Jul 07 '14 at 22:04
Sorry, my previous answer is not correct. I have updated my answer. Could you please try it? please note: use double quotes not single one – Bill Lin Jul 08 '14 at 06:49
Well for the first two lines HADOOP_HOME is undefined (I just put that in the code in my example to hide the actual path in my filesystem). The second two lines work but that just gives the same thing I already had, so I get the same errors – Matthew Saltz Jul 08 '14 at 07:43
can you try to use "org.apache.hadoop.conf.Configuration" somewhere in your java code to see if it compile? I feel it may not – Bill Lin Jul 08 '14 at 07:46
I am; that's where the NoClassDefFound error is coming from. It compiles fine. That's what's so confusing – Matthew Saltz Jul 08 '14 at 09:04

score 0 · Accepted Answer · answered Jul 08 '14 at 09:22

It turned out I just had to add some of the jars from the $HADOOP_HOME/lib directory to the dependencies and it worked. Once I added the dependency for org/apache/commons/logging/LogFactory it cleared up the error for the hadoop Configuration class, and then I just had to add the other required jars later.

NoClassDefFoundError with gradle, giraph, and hadoop

2 Answers2