6

We have an oozie workflow with a shell action that needs more memory than what a map task is given by Yarn by default.

How can we give it more memory?

We have tried adding the following configuration to the action:

<configuration>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>6144</value> <!-- for example -->
  </property>
</configuration>

We have both set this as an inline (in the workflow.xml) configuration and as a jobXml. Neither has had any effect.

Thomas Larsson
  • 697
  • 1
  • 8
  • 17

2 Answers2

19

We found the answer:

A shell action is executed as an oozie "launcher" map task, and this task does not use the normal configuration properties.

Instead you have to prefix the properties with "oozie.launcher" to make them apply to the launcher task.

So in our case, if we use the following configuration for our shell action, it works.

    <configuration>
      <property>
        <name>oozie.launcher.mapreduce.map.memory.mb</name>
        <value>6144</value> <!-- for example -->
      </property>
    </configuration>

This is not obvious from the oozie documentation. We found this here: http://downright-amazed.blogspot.com/2012/02/configure-oozies-launcher-job.html

Indent
  • 4,675
  • 1
  • 19
  • 35
Thomas Larsson
  • 697
  • 1
  • 8
  • 17
0

thank your point, just edit workflow.xml file, add :

<workflow-app name="simple-ONE-wf" xmlns="uri:oozie:workflow:0.1">
    <start to='ONE'/>`enter code here
    <action name="ONE">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapreduce.map.memory.mb</name>
                    <value>4096</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-Xmx3200m</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-XX:MaxPermSize=1g</value>
                </property>
                ...
            </configuration>
           ...
    </action>

    <kill name='kill'>
        <message>Something went wrong: ${wf:errorCode('firstdemo')}</message>
    </kill>
    <end name='end'/>
</workflow-app>
TalkLittle
  • 8,866
  • 6
  • 54
  • 51
kevin lee
  • 19
  • 1
    What's the value added by this answer compared to the (clearer, IMO) existing one posted 3 years ago? – Adriano Repetti Jun 27 '17 at 12:44
  • more details, it is very clearer, can just used, i meet perm OOM, only to add memory is no use, must to add opts and to set -XX:MaxPermSize=1g – kevin lee Jun 27 '17 at 12:48
  • 1
    Wow, if you're targeting Java 7 or earlier. Java 8 has been released even before this question has been asked. Please make it clear in your answer, explain how it's different and where/when it can be used. – Adriano Repetti Jun 27 '17 at 13:16
  • Thank you for this code snippet, which may provide some immediate help. A proper explanation [would greatly improve](//meta.stackexchange.com/q/114762) its educational value by showing *why* this is a good solution to the problem, and would make it more useful to future readers with similar, but not identical, questions. Please [edit] your answer to add explanation, and give an indication of what limitations and assumptions apply. – Toby Speight Jun 30 '17 at 10:40
  • the configuration needs to be at the action section. not at the workflow section. correct? – soMuchToLearnAndShare Nov 29 '19 at 10:41