4

I have a Hadoop job which uses version 1.5 of the commons-codec library. In order to make this job run on EMR AMI 3.x, I had to create a bootstrap action which deleted all earlier versions of the jar from the cluster to prevent them from being loaded instead. These are the relevant lines of that script:

sudo find / -name "commons-codec-1.2.jar" -exec rm -rf {} \;
sudo find / -name "commons-codec-1.3.jar" -exec rm -rf {} \;
sudo find / -name "commons-codec-1.4.jar" -exec rm -rf {} \;

This worked.

But I am now upgrading to AMI 4.x and I have a problem: binary-incompatible earlier versions of the commons-codec jar appear on the slave node's filesystem after the bootstrap script runs.

At the time the bootstrap script runs, the directory /usr/lib/hadoop/lib doesn't exist. But by the time the first step of my job starts, the directory does exist and contains the file /usr/lib/hadoop/lib/commons-codec-1.4.jar. This file is causing a VerifyError because it is loaded instead of the later version of commons-codec which I have bundled in my jar.

Is there a way to delete the incompatible jar before my job starts? Alternatively, is there something I can do to ensure that the correct commons-codec version (which I have bundled in my jar) is loaded instead?

fblundun
  • 987
  • 7
  • 19

1 Answers1

1

You are correct! Starting from emr-4.x.x AMI releases, EMR Applications (Hadoop, Spark, etc.) are installed after Bootstrap Actions are run.

To resolve your issue, you can run your old BA as the first step like below :

yarn node -list|sed -n "s/^\(ip[^:]*\):.*/\1/p" | xargs -t -I{} -P10 ssh -o StrictHostKeyChecking=no -i ~/MyKeyName.pem hadoop@{} "sudo find / -name "commons-codec-1.2.jar" -exec rm -rf {} \;"

to propagate the changes to all slaves - Reference

annunarcist
  • 1,637
  • 3
  • 20
  • 42