0

I would like to run a TeraSort based benchmark test on Hadoop cluster. Script is working, firstly it is in running state, but after a few minutes it stuck in Accepted state and FinalStatus Undefined. I have thought, it is maybe a resource problem so I have modified yarn-site.xml like above.

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>8192</value>
  <description>Amount of physical memory, in MB, that can be allocated for containers.</description>
</property>

<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>2048</value>
</property>

Same problem again. You can see also some graphs above about this process. It stops, when job's progress bar is on ~9-15 %.

Cloudera Manager dashboard

plaidshirt
  • 5,189
  • 19
  • 91
  • 181
  • what is the configuration of your data nodes ( RAM, CUP cores etc?) I think you are setting very high values in configuration and your physical hardware is not up to that configuration. – Ravindra babu Mar 07 '16 at 16:03
  • `RAM: 32 GB - 16 GB CPU: 4 cores - 4 cores Disk: 120 GB - 30 GB` – plaidshirt Mar 07 '16 at 16:14

1 Answers1

2

Please verify the values set for these parameters too:

yarn.scheduler.maximum-allocation-mb, mapreduce.map.memory.mb, mapreduce.map.java.opts, mapreduce.reduce.memory.mb, mapreduce.reduce.java.opts

Start with yarn.scheduler.minimum-allocation-mb to be 512MB. This will help the scheduler to allocate memory to the tasks in smaller increments.

Update 1: This link would help understand a few things: https://www.mapr.com/blog/best-practices-yarn-resource-management

Also, set the input split size as appropriate to your environment. For optimum read performance, your input split size and block size should be same.

Marco99
  • 1,639
  • 1
  • 19
  • 32
  • I have increased these values, but got error message, when trying to runa new job. `MAP capability required is more than the supported max container capability in the cluster. Killing the Job. mapResourceRequest: maxContainerCapability: ` – plaidshirt Mar 07 '16 at 07:39
  • I have used this rule for values: `mapreduce.map.memory.mb < yarn.nodemanager.resource.memory-mb < yarn.scheduler.maximum-allocation-mb` – plaidshirt Mar 07 '16 at 07:45
  • Thanks for link! What do you think, which are optimal values for benchmarking? – plaidshirt Mar 07 '16 at 16:26
  • 1
    It depends on the memory & cpu resources in the machine/cluster. Trial and error would help! – Marco99 Mar 07 '16 at 16:31
  • benchmarking a cluster depends on the resources available in the cluster. Since you are using yarn, it depends on the memory and cpu resources. Tell us the following details. How many nodes you have? What is the available memory on each node? Does nodemanager running on all these nodes? if not, what is the memory and cpu available on the nodes where nodemanager is running? You can get the details of available resources from the Resourcemanager web UI. – Manoj Kumar G Sep 08 '16 at 11:28