How do I get more specific error info on killed job in Oozie

Question

I have a hadoop map-reduce job running as a step in Oozie workflow. It is started using java action which implements org.apache.hadoop.util.Tool.

When the job is being killed for some reason I want to be able to email a notification which should contain the stacktrace if there was an exception during processing.

Currently I do it this way:

<action name="sendErrorNotifications">
    <email xmlns="uri:oozie:email-action:0.1">
        <to>some-dl@company.com</to>
        <subject>Job execution failed ${wf:id()}</subject>
        <body>Job execution failed, error message: [${wf:errorMessage(wf:lastErrorNode())}]</body>
    </email>
    <ok to="fail" />
    <error to="fail" />
</action>

But all I receive is just:

Job execution failed, error message: [Job failed!]

Which is not very useful :) and I need to go and check all the nodes' logs by myself.

How can I get more specific messages? Should I catch my exceptions and wrap into some oozie-catchable one in the Tool, or just use something instead of ${wf:errorMessage...

Thanks

score 2 · Answer 1 · answered Nov 27 '12 at 11:39

2

One suggestion is to catch the exception in your main method, and export a property ('exceptionTrace' for example) with the exception serialized into its value (combined with the capture-output flag), which you can then reference using the wf:actionData('myJavaAction')['exceptionTrace'] EL function.

http://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.7_Java_Action

answered Nov 27 '12 at 11:39

Chris White

29,949
4
71
93

The problem is that it is not possible in Tool to catch an exception that occurred in, say, reducer. All we have there is a job status so we can now that something went wrong... I'll provide the solution I use as a possible answer. – Art Nov 28 '12 at 14:11

score 2 · Accepted Answer · answered Nov 28 '12 at 14:19

I found a way to handle errors and access the cause by using Counters. Maybe it is not what they are designed for, but it seems to be the only way out...

So I catch every Throwable in mapper and reducer like this:

} catch (Throwable t) {
    Counters.Counter counter = reporter.getCounter("Exceptions", t.getClass().getSimpleName());
        counter.increment(1);
    counter.setDisplayName(t.getClass().getSimpleName() + "\n last failed key: " + key.toString() + "\n " + ExceptionUtils.getStackTrace(t));
    reporter.incrCounter("Exceptions", "TOTAL_COUNT", 1);
    reporter.progress();
}

And these counters are easily accessible in the Tool via RunningJob after job is finished. "Exceptions" group contains all exceptions' counters with all needed information in displayName field.

Please comment if you see any problems in this approach or if you know the better one.

How do I get more specific error info on killed job in Oozie

2 Answers2