1

OS: Cent OS 6.4

ISSUE: Installed gmond, gmetad and gweb on a server. Installed spark worker in the same server. configured metrics.properties in $SPARK_HOME/conf/metrics.properties as below...

CONFIGURATION (metrics.properties in spark): 

org.apache.spark.metrics.sink.GangliaSink

host      localhost

port      8649

period    10

unit      seconds

ttl       1

mode      multicast

We are not able to see any metrics in ganglia web. Please do the needful.

-pradeep samudrala

Dharma
  • 3,007
  • 3
  • 23
  • 38
user3928804
  • 1,213
  • 1
  • 8
  • 3

1 Answers1

0

In the first place, those are just indications of the default settings of Ganglia. You should not uncomment that. Taken from the metrics section from the Spark web page (spark page):

To install the GangliaSink you’ll need to perform a custom build of Spark. Note that by embedding this library you will include LGPL-licensed code in your Spark package. For sbt users, set the SPARK_GANGLIA_LGPL environment variable before building. For Maven users, enable the -Pspark-ganglia-lgpl profile. In addition to modifying the cluster’s Spark build user applications will need to link to the spark-ganglia-lgpl artifact.

Mikel Urkia
  • 2,087
  • 1
  • 23
  • 40
  • 1
    Hi Mikel, Thanks for the response. We did build custom spark with LGPL license enabled (export SPARK_GANGLIA_LGPL=true; ./sbt/sbt assembly).All i want to understand is how to enable spark metrics in ganglia gmond conf file, so that we can see the same in gweb utility. – user3928804 Sep 16 '14 at 06:56
  • Hi Mikel, Thanks for the response. Did all the steps mentioned by you. But did not get spark metrics in ganglia. Could you please help us how to enable spark metrics in ganglia. – user3928804 Nov 07 '14 at 04:40