java.lang.IllegalArgumentException: Attempt to add ([custom-jar-with-spark-code].jar) multiple times to the distributed cache

Question

I am trying to run a simple Java Spark job using Oozie on an EMR cluster. The job just takes files from an input path, does few basic actions on it and places the result in different output path.

When I try to run it from command line using spark-submit as shown below, it works fine:

spark-submit --class com.someClassName --master yarn --deploy-mode cluster /home/hadoop/some-local-path/my-jar-file.jar yarn s3n://input-path s3n://output-path

Then I set up the same thing in an Oozie workflow. However, when run from there the job always fails. The stdout log contains this line:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Attempt to add (hdfs://[emr-cluster]:8020/user/oozie/workflows/[WF-Name]/lib/[my-jar-file].jar) multiple times to the distributed cache.
java.lang.IllegalArgumentException: Attempt to add (hdfs://[emr-cluster]:8020/user/oozie/workflows/[WF-Name]/lib/[my-jar-file].jar) multiple times to the distributed cache.

I found a KB note and another question here on StackOverflow that deals with a similar error. But for them, the job was failing due to an internal JAR file - not the one the user is passing to run. Nonetheless, I tried out its resolution steps to remove jar files common between spark & oozie in share-lib and ended up removing a few files from "/user/oozie/share/lib/lib_*/spark". Unfortunately, that did not solve the problem either.

Any ideas on how to debug this issue?

Looks like a bug in the EMR distro. The Spark code base must be "hacked" to fit into an Oozie ShareLib -- many of the init tasks usually done by the Spark launcher are already done by the Oozie launcher (auth tokens, caching etc) hence many Errors must be silenced. Looks like this one was not silenced properly. — Samson Scharfrichter, Apr 18 '19 at 09:07
@SamsonScharfrichter we dont have a dedicated admin for Oozie. Is there a document or so to confirm this is the problem and rectify it? — pallupz, Apr 24 '19 at 13:45
AFAIK, no. Either the AWS tech support gives you a diagnostic, or you compile and debug your own version of Oozie. Cheers... — Samson Scharfrichter, Apr 24 '19 at 14:17

score 1 · Accepted Answer · answered May 02 '19 at 07:08

So we finally figured out the issue - at least in our case.

While creating the Workflow using Hue, when a Spark Action is added, it by default prompts for "File" and "Jar/py name". We provided the path to the JAR file that we wanted to run and the name of that JAR file respectively in those fields and it created the basic action as seen below:

The final XML that it created was as follows:

<action name="spark-210e">
    <spark xmlns="uri:oozie:spark-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>yarn</master>
        <mode>cluster</mode>
        <name>CleanseData</name>
          <class>com.data.CleanseData</class>
        <jar>JCleanseData.jar</jar>
          <spark-opts>--driver-memory 2G --executor-memory 2G --num-executors 10 --files hive-site.xml</spark-opts>
          <arg>yarn</arg>
          <arg>[someArg1]</arg>
          <arg>[someArg2]</arg>
        <file>lib/JCleanseData.jar#JCleanseData.jar</file>        
    </spark>
    <ok to="[nextAction]"/>
    <error to="Kill"/>
</action>

The default file tag in it was causing the issue in our case.

So, we removed it and edited the definition to as seen below and that worked. Note the change to <jar> tag as well.

<action name="spark-210e">
    <spark xmlns="uri:oozie:spark-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>yarn</master>
        <mode>cluster</mode>
        <name>CleanseData</name>
          <class>com.data.CleanseData</class>
        <jar>hdfs://path/to/JCleanseData.jar</jar>
          <spark-opts>--driver-memory 2G --executor-memory 2G --num-executors 10 --files hive-site.xml</spark-opts>
          <arg>yarn</arg>
          <arg>[someArg1]</arg>
          <arg>[someArg1]</arg>
    </spark>
    <ok to="[nextAction]"/>
    <error to="Kill"/>
</action>

PS: We had a similar issue with Hive actions too. The hive-site.xml file we were supposed to pass with Hive action - which created a <job-xml> tag - was also causing issues. So we removed it and it worked as expected.

java.lang.IllegalArgumentException: Attempt to add ([custom-jar-with-spark-code].jar) multiple times to the distributed cache

1 Answers1