We are using AWS EMR to run spark jobs. From ganglia we see that the memory utilisation of our cluster is low as compared to the allocated memory. This is the case with cpu utilisation as well.
We are currently reporting spark metrics by implementing the Spark listener. But I can’t find how to get ganglia metrics out from the spark code. Is this information available anywhere in spark events as well?
You can try sparklens (https://github.com/qubole/sparklens) to first find out the right executor count for your spark application and then configure the cluster accordingly. Cluster utilisation depends on application scalability characteristics also.
To get spark JMX metrics
1) Create a metrics.properties file in spark/conf/
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource master.source.jvm.class=org.apache.spark.metrics.source.JvmSource worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
2) Download jolokai agent jar to some location in your cluster. Other JMX agents should work as well.
3) Update the following Spark Overrides:
spark.metrics.conf spark/conf/metrics.properties spark.driver.extraJavaOptions -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/tmp/jolokia-jvm.jar=port=8779,host=localhost spark.executor.extraJavaOptions -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/tmp/jolokia-jvm.jar=port=8781,host=localhost"
4) Use telegraph or other monitoring tools to get metrics from the agents.
