Pig : Container is running beyond physical memory limits in cdh 5 using oozie

Question

I'm trying to run a simple pig script which running f9 in grunt shell but not working with oozie ,getting error like this:

Container [pid=2617,containerID=container_1438923434512_12103_01_000002] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.9 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1438923434512_12103_01_000002..

actually i'm calling a shell script through oozie which intern calls pig script and getting error like that.

How could i make it workable in oozie

Answer 1

Looks like the default for YARN container size on that cluster (1 GB of RAM) is way too low for your job.

But it's not clear whether the YARN error shown relates to the Shell action (running grunt) or to a child MapReduce execution.

Plan A - Assuming that it's the child MapReduce execution that requires more RAM, on top of the grunt script just add

set mapreduce.map.memory.mb    5120
set mapreduce.reduce.memory.mb 5120

Reference: Pig documentation, YARN documentation

Plan B - If it is actually the Shell action that requires more RAM, add some configuration in workflow.xml

<action name="PiggyBack">
    <shell xmlns="uri:oozie:shell-action:0.2">
      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>

      <configuration>
         <property>
            <name>oozie.launcher.mapreduce.map.memory.mb</name>
            <value>5120</value>
         </property>
      </configuration>

etc.

Reference: I found the hint myself a few months back from this question so I suggest you give it a +1 if it was helpful...