8

I want to try about setting memory in YARN, so I'll try to configure some parameter on yarn-site.xml and mapred-site.xml. By the way I use hadoop 2.6.0. But, I get an error when I do a mapreduce job. It says like this :

15/03/12 10:57:23 INFO mapreduce.Job: Task Id :
attempt_1426132548565_0001_m_000002_0, Status : FAILED
Error: Java heap space
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

I think that I have configured it right, I give map.java.opts and reduce.java.opts the small size = 64 MB. I've try to configure some parameter then, like change the map.java.opts and reduce.java.opts on mapred-site.xml, and I still get this error. I think that I do not really understand how YARN memory works. BY the way for this I try on single node computer.

trincot
  • 317,000
  • 35
  • 244
  • 286
Kenny Basuki
  • 625
  • 4
  • 11
  • 27
  • 64MB is very small. Try 1GB as a starting point. – Jeremy Beard Mar 13 '15 at 04:11
  • On your mind, what causing this error? I set 64 MB because my computer memory is very small. 64 MB is for java process right, should it's smaller than yarn.app.mapreduce.am.command-opts or yarn.app.mapreduce.am.resource.mb. Please give me an explanation...Thanks a lot – Kenny Basuki Mar 13 '15 at 05:59

2 Answers2

11

Yarn handles resource management and also serves batch workloads that can use MapReduce and real-time workloads.

There are memory settings that can be set at the Yarn container level and also at the mapper and reducer level. Memory is requested in increments of the Yarn container size. Mapper and reducer tasks run inside a container.

mapreduce.map.memory.mb and mapreduce.reduce.memory.mb

above parameters describe upper memory limit for the map-reduce task and if memory subscribed by this task exceeds this limit, the corresponding container will be killed.

These parameters determine the maximum amount of memory that can be assigned to mapper and reduce tasks respectively. Let us look at an example: Mapper is bound by an upper limit for memory which is defined in the configuration parameter mapreduce.map.memory.mb.

However, if the value for yarn.scheduler.minimum-allocation-mb is greater than this value of mapreduce.map.memory.mb, then the yarn.scheduler.minimum-allocation-mb is respected and the containers of that size are given out.

This parameter needs to be set carefully and if not set properly, this could lead to bad performance or OutOfMemory errors.

mapreduce.reduce.java.opts and mapreduce.map.java.opts

This property value needs to be less than the upper bound for map/reduce task as defined in mapreduce.map.memory.mb/mapreduce.reduce.memory.mb, as it should fit within the memory allocation for the map/reduce task.

Gaurav Mishra
  • 1,009
  • 6
  • 11
  • Thanks a lot mr gaurav for your help. By the way, I want to ask again, is the number of map container according to the map task number? For example the map task number is 7, so if I also have 5 map container? Is that true if my map container after finished single map task will do another map task without create a new container? Thanks a lot.... – Kenny Basuki Mar 15 '15 at 08:08
  • Hi Kenny, I didn't get your question. Could you elaborate it. – Gaurav Mishra Mar 15 '15 at 21:26
  • Hi kenny, please go through this link it may help you. http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ – Gaurav Mishra Mar 16 '15 at 11:43
  • 1
    you forgot to mention the source of your (?) answer https://blogs.msdn.microsoft.com/bigdatasupport/2014/11/11/some-commonly-used-yarn-memory-settings/ – AdamSkywalker Feb 08 '18 at 13:32
2

What @Gaurav said is correct. I had similar issue,i tried some thing like below.Include below properties in yarn-site.xml and restart VM

<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for    containers</description>
</property>

<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>
Nagendra Prasad Balaka
  • 549
  • 3
  • 9
  • 26