0

I was wondering if there's a more detailed version of such spark webpage Monitoring metrics

Actually I'm using grafana and this is my dashboard, I'm not totally sure if we can assume the Heap usage as RAM usage.

Grafana dashboard

What I would like to achieve is to show the RAM used from each node and the total RAM available for each of them.

Any suggestion? I have a lot of different metrics I cannot report them all but we could continue the conversation in the comments (and then I'll add it in this topic for sake of clearance for the next developers interested)

I need help, monitoring metrics are not so well explained

callme_fantastique
  • 11
  • 3
  • Can you provide more details, like which environment you use *ApacheSpark* module when running PySpark scripts? There are some cloud-based envis such as [Databricks](https://azure.microsoft.com/en-us/products/databricks/), which offer users to monitor the trend of allocation of recourses/running-threads as well as RAM in distributed systems when one runs the notebook. – Mario Jan 26 '23 at 15:42
  • Hi Mario sorry for the late reply, actually I've created on a VM a local standalone spark cluster made of 1 driver and 2 workers and I submit spark app through spark-submit. No I'm not using Databricks (I know it presents Ganglia for showing metrics) – callme_fantastique Feb 03 '23 at 16:19
  • Usually, cluster-based envi provide users with monitoring tools during the computation for resource consumption of (RAM) for the driver or distributed-based workers, along with the progress of the run with percentage metrics. I'm not familiar with _Grafana_. Have you checked this [post](https://stackoverflow.com/questions/73493136/grafana-how-to-display-ram-by-user-service?rq=1)? – Mario Feb 03 '23 at 16:28

0 Answers0