After running my spark application, I want to monitor its memory and cpu usage to evaluate its performance but couldn't find any option. Is it possible to monitor it? How can I monitor memory and CPU usage by spark application?
Asked
Active
Viewed 2,221 times
1
-
1Maybe [this](https://spark.apache.org/docs/latest/monitoring.html#web-interfaces) can help you? – Alexandre Dupriez Oct 23 '17 at 08:40
1 Answers
2
There are a few options:
- Ganglia is one
- If you're running on your own cluster, HDP or Cloudera both have real time CPU & memory consumption charts.
- If you want specific JVM metrics, then I'd recommend FlameGraph, though it's not real time.
- There's also Grafana, it's extremely powerful, you can track many metrics with it, and it's real time.
moon
- 1,702
- 3
- 19
- 35