7
<property>
<name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name>
<value>false</value>
</property>

I want to override the above property to true. The property needs to be false for the rest of the jobs on the cluster, but I need, in my oozie workflow, hadoop to create _SUCCESS file in the output directory after the completion of job. Its a hive action in the workflow which writes output. Please help.

Bhargav
  • 111
  • 1
  • 7

2 Answers2

9

Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

see 

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.

Daniel Darabos
  • 26,991
  • 10
  • 102
  • 114
Matthew Rathbone
  • 8,144
  • 7
  • 49
  • 79
2

You can add 'dfs' command to your hive script, like

dfs -touchz '$table_base_path'/dt='${partition}'/_SUCCESS

https://archive.cloudera.com/cdh4/cdh/4/hive/language_manual/cli.html

Doroshenko
  • 111
  • 1
  • 3