2

I am using CDH (Cloudera Hadoop) version 5.12.0 (which uses: Hadoop 2.6.0 Oozie 4.1.0) and Scalding 2.11

I am using a shaded jar with my dependencies built in.

I can run all my jobs properly without any error using a hadoop jar command as such:

hadoop jar /<path>/<to>/<my>/<project>.jar com.twitter.scalding.Tool 
    -libjars <comma>,<separated>,<list>,<of>,<dependencies>
    -D mapreduce.framework.name=yarn 
    -D yarn.app.mapreduce.am.staging-dir=/<path>/<to>/<staging>/<dir>
    -D mapreduce.map.output.compress=True 
    <my>.<scalding>.<job> --hdfs --input <my>/<input> --output <output>/<dir>

I can also run an oozie workflow with pig actions or other actions without any trouble. But when I try to run an oozie workflow with java actions that call scalding like so:

<action name="myAction">
<java>
  <job-tracker>${jobTracker}</job-tracker>
  <name-node>${nameNode}</name-node>
  <prepare>
    <delete path="<output>/<dir>"/>
  </prepare>
  <configuration>
    <property>
      <name>oozie.action.sharelib.for.java</name>
      <value>scalding</value>
    </property>
    <property>

   <name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
      <value>true</value>
    </property>
    <property>
      <name>oozie.launcher.mapreduce.task.classpath.first</name>
      <value>true</value>
    </property>
    <property>
      <name>oozie.launcher.mapreduce.job.user.classpath.first</name>
      <value>true</value>
    </property>
    <property>
      <name>mapreduce.job.user.classpath.first</name>
      <value>true</value>
    </property>
    <property>
      <name>mapreduce.task.classpath.user.precedence</name>
      <value>true</value>
    </property>
    <property>
      <name>mapreduce.task.classpath.first</name>
      <value>true</value>
    </property>
    <property>
      <name>mapreduce.map.log.level</name>
      <value>DEBUG</value>
    </property>
    <property>
      <name>mapreduce.reduce.log.level</name>
      <value>DEBUG</value>
    </property>

    <property>
      <name>mapreduce.job.log4j-properties-file</name>
      <value>${appLocation}/conf</value>
    </property>

    <property>
      <name>mapreduce.task.files.preserve.failedtasks</name>
      <value>true</value>
    </property>

    <property>
      <name>mapreduce.job.jvm.numtasks</name>
      <value>-1</value>
    </property>

    <property>
      <name>oozie.mapreduce.uber.jar</name>
      <value>/<path>/<to>/<my>/<class>.jar</value>
    </property>


  </configuration>
  <main-class>com.twitter.scalding.Tool</main-class>
  <java-opts>-Xms2G -Xmx3G</java-opts>
  <arg><my>.<scalding>.<job></arg>
  <arg>--hdfs</arg>
  <arg>--input</arg>
  <arg><my>/<input></arg>
  <arg>--output</arg>
  <arg><output>/<dir></arg>
  <file>/<path>/<to>/<my>/<project>.jar</file>
</java>

<ok to="end" />
<error to="sendEmailFailure" />

I have also set:

oozie.libpath=<path>/<to>/<lib>/<directory>
oozie.use.system.libpath=true

in my properties file and all the necessary jars are within the libpath, but I get errors where the java action can't find the dependencies it needs:

Error: java.io.IOException: Split class cascading.tap.hadoop.io.MultiInputSplit not found at 
org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363) at 
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.io.MultiInputSplit not found at

org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2108) at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361) ... 7 more

This happens inconsistently, in that if I only put one action in the workflow, sometimes the workflow completes, sometimes it fails. If I put more than one action in the workflow, it definitely fails on some action eventually, not always the same action but always with the same error.

I believe the problem is coming from the way in which the action is getting it's dependencies and sometimes it loads them in the correct order and gets the one I want first and sometimes it doesn't and is missing the dependency it needs. But I'm new to oozie, so who knows?

I think I can increase the number of max attempts for each action taken by oozie, but I feel like that is not really a solution and is sort of akin to rolling the dice against the cluster.

I saw people post this issue a while back and rolling back to an older verison of CDH (4.1) but that's not an option for me.

Suggestions?

Murium
  • 183
  • 7

0 Answers0