1

In a YARN 2.6.0 cluster, is there a way to be able to get all the connected node's CPU utilization at the ResourceManager? Also, is the source code modifiable such that we can decide the nodes for a map-reduce job based on the utilization. If yes, where would this change take place?

Яois
  • 3,838
  • 4
  • 28
  • 50
Abhishek Sarangan
  • 11
  • 1

1 Answers1

0

Pls, find the implementation of Container Monitor:(CPU Utilization)

hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java

We have methods to check if a container is over the limitation.

isProcessTreeOverLimit will show you how yarn get the memory usage of certain container(process).

hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java

The above file shows you how Yarn gets memory usage: tracking process file in/proc.

Partha Kaushik
  • 690
  • 4
  • 8