2

Say I have a EMR job running on 11 node cluster: m1.small master node while 10 m1.xlarge slave nodes.

Now one m1.xlarge node has 15 GB of RAM.

How to then decide on the number of parallel mappers and reducers which can be set?

My jobs are memory intensive and I would like to have more and more of heap allotted to JVM.

Another related question: If we set the following parameter:

 <property><name>mapred.child.java.opts</name><value>-Xmx4096m</value></property>
 <property><name>mapred.job.reuse.jvm.num.tasks</name><value>1</value></property>
 <property><name>mapred.tasktracker.map.tasks.maximum</name><value>2</value></property>
 <property><name>mapred.tasktracker.reduce.tasks.maximum</name><value>2</value></property>

So will this 4GB be shared by 4 processes (2 mapper and 2 reducer) or will they all get 4GB each?

Amar
  • 11,930
  • 5
  • 50
  • 73

1 Answers1

2

They will each get 4gb.

You should check what your heap setting is for the task trackers and the data nodes, then you'll have an idea of how much memory you have left over to allocate to children (the actual mappers / reducers).

Then it's just a balancing act. If you need more memory, you'll want less mappers / reducers, and vice versa.

Also try to keep in mind how many cores your CPU has, you don't want 100 map tasks on a single core. To tweak, it's best to monitor both heap usage and cpu utilization over time so you can fiddle with the knobs.

Matthew Rathbone
  • 8,144
  • 7
  • 49
  • 79
  • Ok, so for m1.xlarge we have 15 GB of RAM, but I have say 2 mappers and 2 reducers, then it will total to 16 GB, is it bad idea to do like this and have it on the edge? – Amar Nov 07 '12 at 17:09
  • try it and see :-) If you get OOME then you'll have to lower the heap space. At this point it's trial and error. A good way to burn-test a cluster setup is to use the contrib tera-sort mapreduce job, that way you have something to benchmark runtime against. You might also want to update the io.sort.mb to a higher value given how much ram you have. – Matthew Rathbone Nov 07 '12 at 19:30