4

I would like my map and reduce tasks to run in parallel. However, despite trying every trick in the bag, they are still running sequentially. I read from How to set the precise max number of concurrently running tasks per node in Hadoop 2.4.0 on Elastic MapReduce, that using the following formula, one can set the number of tasks running in parallel.

min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb, 
 yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcores)

However, I did that, as you can see from the yarn-site.xml and mapred-site.xml I am using below. But the tasks still run sequentially. Note that I am using the open source Apache Hadoop and not Cloudera. Would shifting to Cloudera solve the problem? Also note that my input files are big enough that dfs.block.size should also not be an issue.

yarn-site.xml

    <configuration>
    <property>
      <name>yarn.nodemanager.resource.memory-mb</name>
      <value>131072</value>
    </property>
    <property>
      <name>yarn.nodemanager.resource.cpu-vcores</name>
      <value>64</value>
    </property>
    </configuration>

mapred-site.xml

    <configuration>
    <property>
      <name>mapred.job.tracker</name>
      <value>localhost:9001</value>
    </property>

    <property>
      <name>mapreduce.map.memory.mb</name>
      <value>16384</value>
    </property>

    <property>
      <name>mapreduce.reduce.memory.mb</name>
      <value>16384</value>
    </property>

    <property>
        <name>mapreduce.map.cpu.vcores</name>
        <value>8</value>
    </property>

    <property>
        <name>mapreduce.reduce.cpu.vcores</name>
        <value>8</value>
    </property>
    </configuration>
Community
  • 1
  • 1
MetallicPriest
  • 29,191
  • 52
  • 200
  • 356
  • changing the hadoop distribution would not make any sense..the issue could be somewhere else.. – suresiva Apr 30 '15 at 11:14
  • can you let me know the number of nodes and containers in your cluster?.. use `http://:8088/cluster/nodes` to check... – suresiva Apr 30 '15 at 11:15
  • @hserus: I have one node, 64 virtual cores. And I don't really know what you mean about containers. How can I check the number of containers? – MetallicPriest Apr 30 '15 at 11:21
  • ok...are u able to see this resource manager's page `http://:8088/cluster/nodes` ?... you would be seeing one node listed...just let me know the containers, mem avail, mem used values showed against the node... – suresiva Apr 30 '15 at 11:26
  • @hserus: Is there any way to see this without a web browser. Because the server on which I'm running this has got none, and I don't have admin rights either. – MetallicPriest Apr 30 '15 at 11:28
  • ok, no issues...atleast let me know the free memory value from the result of `free -g` on the node's shell. – suresiva Apr 30 '15 at 11:33
  • From free -g, free memory is 11. Meanwhile total is 156, used 145, shared 76 and cached 139. – MetallicPriest Apr 30 '15 at 11:36
  • How many map and reduce tasks are spawning for your job? What exactly do you mean by "I would like my map and reduce tasks to run in parallel"? Do you mean that you want multiple map tasks running concurrently? How do you know they're running "sequentially"? You don't need a browser on the server. As long as port 8088 is open you can point any browser at the url above. – Pradeep Gollakota Apr 30 '15 at 18:50

1 Answers1

5

Container is the logical execution template reserved for the execution of Map/Reduce tasks on every node of the culster.

The yarn.nodemanager.resource.memory-mb property tells the YARN resource manager to reserve that much of ram memory for all containers to be dispatched in the node to execute Map/Reduce tasks. This is the maximum upper bound of the memory will be reserved for every container.

But in you case, the free memory in the node is almost 11GB, and you have configured yarn.nodemanager.resource.memory-mb to almost 128GB(131072) , mapreduce.map.memory.mb & mapreduce.reduce.memory.mb as 16GB . The required upper bound size for Map/Reduce containers is 16Gb wich is higher than 11GB of the free memory* . This could be a reason that you were allocated only one container in the node for execution.

We shall reduce the value of mapreduce.map.memory.mb , mapreduce.reduce.memory.mb properties than the value of free memory to get more than one container running in parallel.

Also see some ways to increase the free memory since its already more 90% of it used.

Hope this helps :) ..

suresiva
  • 3,126
  • 1
  • 17
  • 23
  • Well I told you the reading in the first row displayed by free -g. I think the actual free and unused memory is listed in the second row. It is 150 GB in my case. So, yea, I've got plenty :). So, what parameters would you suggest for this amount of memory? – MetallicPriest Apr 30 '15 at 12:18
  • oh yes, the free memory in the second row is the actual free including the buffer/cache...hmm, but not sure which will be picked by YARN...but we can experiment it... – suresiva Apr 30 '15 at 12:50
  • apologies..i was little wrong, so edited the answer :( pls check..For a try, we shall keep the `yarn.nodemanager.resource.memory-mb` value as it is and shall try with 2048MB for `mapreduce.map.memory.mb` and `mapreduce.reduce.memory.mb` to parallel get containers executed. kindly share your observations.. You dont have to update mapred-site.xml, instead these two properties can be updated in the job configuration of the mapreduce code itself. – suresiva Apr 30 '15 at 13:33
  • i tried to see how jvm spawns on memory.Actually jvm takes the memory out of the free memory shown from the second line. anyhow can you try above configurations and share your observations. we could be in wrong direction, we shall try others after seeing the observations. – suresiva Apr 30 '15 at 14:24
  • What is the method to change mapreduce.map.memory.mb and mapreduce.reduce.memory.mb in the code? – MetallicPriest Apr 30 '15 at 15:15
  • Anyway I have tried these things, but still no luck! – MetallicPriest Apr 30 '15 at 15:30