0

How do I sink Spark Streaming metrics to this StatsD sink for executors?

Similar to other reported issues (sink class not found, sink class in executor), I can get driver metrics, but executors throw ClassNotFoundException with my setup:

  • StatsD sink class is compiled with my Spark-Streaming app (my.jar)

  • spark-submit is run with:

    • --files ./my.jar (to pull jar containing sink into executor)
    • --conf "spark.executor.extraClassPath=my.jar"

  • Spark Conf is configured in the driver with:

    val conf = new SparkConf()

conf.set("spark.metrics.conf.*.sink.statsd.class",
         "org.apache.spark.metrics.sink.StatsDSink")
    .set("spark.metrics.conf.*.sink.statsd.host", conf.get("host"))
    .set("spark.metrics.conf.*.sink.statsd.port", "8125")
Ihor Kaharlichenko
  • 5,944
  • 1
  • 26
  • 32
ecoe
  • 4,994
  • 7
  • 54
  • 72

2 Answers2

2

Looks you hit the bug https://issues.apache.org/jira/browse/SPARK-18115. I hit it too and googled your question :(

gengmao
  • 113
  • 1
  • 6
1

Copy your jar files to the $SPARK_HOME/jars folder.

Tipx
  • 7,367
  • 4
  • 37
  • 59
asahin
  • 29
  • 2
  • Thanks. Succeed on YARN `spark-submit` job. But could you explain why? Why is not enough run with parameters-`--files` or `extraClassPath`? – nullmari Apr 17 '19 at 10:26