1

I need to build a chart showing me how much pending (ACCEPTED) Hadoop tasks I have on a cluster. To be more detailed, I need to get this value:

yarn application -list -appStates ACCEPTED | grep ACCEPTED | wc -l

every N seconds, and build the real time graph for monitoring.

Instead of whiting charting code "from scratch" I have chosen Cloudera Manager with its Chart Builder developed to simplify creating such charts. But in the list of Cloudera metrics I can't find any metric which would help me to build graph (except apps_ingested_rate and active_applications_across_yarn_pools but they always show 0). So, does it exist some way to build chart for monitoring quantity of pending YARN applications using Cloudera Manager?

Version of Cloudera Manager: 5.11 free.

VeLKerr
  • 2,995
  • 3
  • 24
  • 47

2 Answers2

1

I don't know if this information is available in Cloudera Manager -- since I'm using HDP -- but one of the easiest way to get this information is to use the YARN ResourceManager REST API’s.

The number of pending applications is directly exposed by the Cluster Metrics API end point in the appsPending field.

GET http://<rm http address:port>/ws/v1/cluster/metrics

# {
#   "clusterMetrics":
#  {
#    "appsSubmitted":0,
#    "appsCompleted":0,
#    "appsPending":0,
Romain
  • 19,910
  • 6
  • 56
  • 65
0

I'm not using the free version but you should be able to build a chart using "select apps_pending".

Perhaps the reason you don't see anything is you don't have enough jobs that saturate your cluster so apps pending is always zero. To test you can run some jobs and use "select apps_submitted_rate" to verify you are seeing data correctly.

tk421
  • 5,775
  • 6
  • 23
  • 34