How to retrieve max containers available from YARN in Hadoop Cluster?

Question

How do i get the total count i.e max capacity of containers from Resource Manager of YARN in Hadoop Cluster. I did try REST Api's and JMX metrics am getting only total allocated containers and pending containers.So is there any way i can retrieve the max containers that can be created count?

Answer 1

In YARN, we have minimum allocation property, so all container have minimuum of that value. So based on that you can caluculate maximum number of containers that can be launched (divide by total memory available for yarn). I think this is the only way to get maximum containers that can be launched.

So in your programs, get these two properties and doing some math will get you required result.

Properties to look for are :

    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>1024</value>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>8192</value>

Answer 2

The RM API cluster metrics shows you total vcores and total memory.

https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Metrics_API

So if you do a GET on RM-URL/ws/v1/cluster/metrics you'd see the values of "totalMB" and "totalVirtualCores".