7

I am evaluating YARN for a project. I am trying to get the simple distributed shell example to work. I have gotten the application to the SUBMITTED phase, but it never starts. This is the information reported from this line:

ApplicationReport report = yarnClient.getApplicationReport(appId);

Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = DEFAULT_PARTITION; AM Resource Request = memory:1024, vCores:1; Queue Resource Limit for AM = memory:0, vCores:0; User AM Resource Limit of the queue = memory:0, vCores:0; Queue AM Resource Usage = memory:128, vCores:1;

The solutions for other developers seems to have to increase yarn.scheduler.capacity.maximum-am-resource-percent in the yarn-site.xml file from its default value of .1. I have tried values of .2 and .5 but it does not seem to help.

avariant
  • 2,234
  • 5
  • 25
  • 33
sealfeeder
  • 71
  • 1
  • 1
  • 4

3 Answers3

8

Looks like you did not configure the RAM allocated to Yarn in a proper way. This can be a pin in the ..... if you try to infer/adapt from tutorials according to your own installation. I would strongly recommend that you use tools such as this one: 

wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
rm hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
mv hdp_manual_install_rpm_helper_files-2.6.0.3.8/ hdp_conf_files
python hdp_conf_files/scripts/yarn-utils.py -c 4 -m 8 -d 1 false
  • -c number of cores you have for each node
  • -m amount of memory you have for each node (Giga)
  • -d number of disk you have for each node
  • -bool "True" if HBase is installed; "False" if not

This should give you something like:

Using cores=4 memory=8GB disks=1 hbase=True
Profile: cores=4 memory=5120MB reserved=3GB usableMem=5GB disks=1
Num Container=3
Container Ram=1536MB
Used Ram=4GB
Unused Ram=3GB
yarn.scheduler.minimum-allocation-mb=1536
yarn.scheduler.maximum-allocation-mb=4608
yarn.nodemanager.resource.memory-mb=4608
mapreduce.map.memory.mb=1536
mapreduce.map.java.opts=-Xmx1228m
mapreduce.reduce.memory.mb=3072
mapreduce.reduce.java.opts=-Xmx2457m
yarn.app.mapreduce.am.resource.mb=3072
yarn.app.mapreduce.am.command-opts=-Xmx2457m
mapreduce.task.io.sort.mb=614

Edit your yarn-site.xml and mapred-site.xml accordingly.

 nano ~/hadoop/etc/hadoop/yarn-site.xml
 nano ~/hadoop/etc/hadoop/mapred-site.xml

Moreover, you should have this in your yarn-site.xml

<property>
      <name>yarn.acl.enable</name>
      <value>0</value>
</property>

<property>
       <name>yarn.resourcemanager.hostname</name>
       <value>name_of_your_master_node</value>
</property>

<property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
</property>

and this in your mapred-site.xml:

<property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
</property>

Then, upload your conf files to each node using scp (If you uploaded you ssh keys to each one)

for node in node1 node2 node3; do scp ~/hadoop/etc/hadoop/* $node:/home/hadoop/hadoop/etc/hadoop/; done

And then, restart yarn

stop-yarn.sh
start-yarn.sh

and check that you can see your nodes:

hadoop@master-node:~$ yarn node -list
18/06/01 12:51:33 INFO client.RMProxy: Connecting to ResourceManager at master-node/192.168.0.37:8032
Total Nodes:3
     Node-Id         Node-State Node-Http-Address   Number-of-Running-Containers
 node3:34683            RUNNING        node3:8042                              0
 node2:36467            RUNNING        node2:8042                              0
 node1:38317            RUNNING        node1:8042                              0

This might fix the issue (good luck) (additional info)

zar3bski
  • 2,773
  • 7
  • 25
  • 58
  • An useful setting is `yarn.nodemanager.resource.detect-hardware-capabilities` as `true`, see https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml for more info (replace `3.2.0` in this URL with your Hadoop version). – Eugene Gr. Philippov Feb 02 '19 at 16:32
1

I got the same error and tried to solve it hard. I realized the resource manager had no resource to allocate the application master (AM) of the MapReduce application.
I navigated on browser http://localhost:8088/cluster/nodes/unhealthy and examined unhealthy nodes (in my case there was only one) -> health report. I saw the warning about that some log directories filled up. I cleaned those directories then my node became healthy and the application state switched to RUNNING from ACCEPTED. Actually, as a default, if the node disk fills up more than %90, YARN behaves like that. Someway you have to clean space and make available space lower than %90. My exact health report was:

1/1 local-dirs usable space is below configured utilization percentage/no more usable space [ /tmp/hadoop-train/nm-local-dir : used space above threshold of 90.0% ] ; 
1/1 log-dirs usable space is below configured utilization percentage/no more usable space [ /opt/manual/hadoop/logs/userlogs : used space above threshold of 90.0% ]
Erkan Şirin
  • 1,935
  • 18
  • 28
0

Add below properties to yarn-site.xml and restart dfs and yarn

<property>
   <name>yarn.scheduler.capacity.root.support.user-limit-factor</name>  
   <value>2</value>
</property>
<property>
   <name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>
   <value>0.0</value>
</property>
<property>
   <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
   <value>100.0</value>
</property>
Sunil N
  • 41
  • 3