1

I have a jar that uses the Hadoop API to launch various remote mapreduce jobs (ie, im not using the command-line to initiate the job). The service jar that executes the various jobs is built with maven's "jar-with-dependencies".

My jobs all run fine except one that uses commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;

I think this is because my jar is including commons-codec 1.7 whereas my Hadoop install's lib has commons-codec 1.4 ...

Is their any way to instruct Hadoop to use the distributed commons-codec 1.7 (I assume this is distributed as a job dependency) rather than the commons-codec 1.4 in the hadoop 1.0.3 core lib?

Many thanks!

Note: Removing commons-codec-1.4.jar from my Hadoop library folder does solve the problem, but doesn't seem too sane. Hopefully there is a better alternative.

Aiden Bell
  • 28,212
  • 4
  • 75
  • 119

1 Answers1

0

Two approaches:

  • You should be able to exclude commons-codec from within the hadoop dependency and add another explicit dependency for commons-codec
  • Try setting the scope to provided so that none of the hadoop jars get included. This assumes that those jars would be in the runtime class path.
Chris Gerken
  • 16,221
  • 6
  • 44
  • 59
  • Thanks for the answer, not 100% sure what you mean about the first point and the second point, my hadoop client libraries in the pom are already set to provided scope. – Aiden Bell Oct 02 '12 at 20:38
  • Within the dependency block in a pom you can list a number of implied dependencies that should be excluded from the maven action. So if version 1.4 is the indirectly implied version you can exclude that and explicitly include a new dependency for 1.7 in your pom. – Chris Gerken Oct 02 '12 at 21:11
  • Yep, that is already being done. commons-codec 1.7 is added to the pom with hadoop libs being provided. The problem is when hadoop enters the MR after distributing my jar to the job tracker. – Aiden Bell Oct 03 '12 at 11:01