8

I'm using hadoop-2.7.2 and I did a MapReduceJob with IntelliJ. In my job, I'm using apache.commons.cli-1.3.1 and I put the lib in the jar.

When I use the MapReduceJob on my Hadoop cluster I have a NoSuchMethodError:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;

I don't understand because the method exist in the class Option and the class Option is extracted from the commons-cli.jar to my application jar. Moreover, I don't have this issue with my others libraries.

Thank you for your time.

Vladimir Vagaytsev
  • 2,871
  • 9
  • 33
  • 36
Antonin
  • 81
  • 1
  • 2
  • 1
    I just found that hadoop use commons-CLi 1.2. I think this is the source of my issue but I don't know how to fix it. – Antonin Jul 29 '16 at 09:49
  • You can try to exclude `commons-cli-1.2` from Hadoop dependency in your `pom.xml`, then Hadoop willl use your `commons-cli-1.3.1`. If it causes any errors, you'd better use `1.2` in your code. – Vladimir Vagaytsev Jul 29 '16 at 12:04
  • I did something like that but my boss doesn't want this way because it doesn't fix the issue. If we have the same issue later with another lib, he doesn't want to fix the issue again. But thank you for your advice. – Antonin Jul 29 '16 at 14:09
  • I fixed it by changing the sequence in which the jars are declared in the classpath. Please check my answer. – Gyanendra Dwivedi Jul 06 '17 at 19:23

3 Answers3

5

We were able to fix this error using maven class relocations. If you are using the shade plugin to build your jar add following to pom.xml under appropriate section:

<!-- necessary to fix NoSuchMethodError: org.apache.commons.cli.Option.builder -->
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.commons.cli</pattern>
                                    <shadedPattern>org.shaded.commons.cli</shadedPattern>
                                </relocation>
                            </relocations>

Also an explicit reference to v1.3+ of commons-cli needs to be added at the TOP of the dependencies section before any dependency that may have a transitive reference to an older version of commons-cli.

morpheus
  • 18,676
  • 24
  • 96
  • 159
3

The problem seems to be related to how the classloader is loading the classes. Because the static Builder class was in common-cli 1.4, while some of the hadoop dependencies were still referring to older version - the issue occurred.

In my case, the issue resolved by changing the sequence of jar file addition into the classpath in the shell script responsible for setting up environment before program execution. Earlier, I was adding the jar into classpath like

CLASSPATH=<Hadoop Jars>:<Common CLI jar>:$CLASSPATH

is changed to

CLASSPATH=<Common CLI jar>:<Hadoop Jars>:$CLASSPATH

and it fixed the problem.

Gyanendra Dwivedi
  • 5,511
  • 2
  • 27
  • 53
1

We solved this issue with the next gradle configuration:

 compile('org.apache.parquet:parquet-tools:1.9.0'){
  exclude module:"commons-cli"
 }
Evgeny Konurbaev
  • 469
  • 4
  • 6