NoSuchMethodError when running Spark streaming example with kafka

Question

I'm using spark-2.4.4-bin-without-hadoop, and I wanna test the self-contained example JavaDirectKafkaWordCountexample.

From the official document, it mentioned the application should include spark-streaming-kafka-0-10_2.12 this dependency. So I download the spark-streaming-kafka-0-10_2.12-2.4.0.jar to jars directory.

However, When I run run-example streaming.JavaDirectKafkaWordCount device1:9092 group_id topic, it manifests NoSuchMethodError:

20/01/13 11:51:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1549bba7{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
        at org.apache.spark.streaming.kafka010.PreferConsistent$.<init>(LocationStrategy.scala:42)
        at org.apache.spark.streaming.kafka010.PreferConsistent$.<clinit>(LocationStrategy.scala)
        at org.apache.spark.streaming.kafka010.LocationStrategies$.PreferConsistent(LocationStrategy.scala:66)
        at org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent(LocationStrategy.scala)
        at org.apache.spark.examples.streaming.JavaDirectKafkaWordCount.main(JavaDirectKafkaWordCount.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/01/13 11:51:12 INFO spark.SparkContext: Invoking stop() from shutdown hook

Your error seems to suggest your are using Spark with Scala 2.11, not 2.12. Please fix your packages — OneCricketeer, Jan 13 '20 at 06:31
yes, thanks for your remind. I just figure out how to solve it in a simple way: download this `spark-streaming-kafka-0-10_2.11` instead. — DennisLi, Jan 13 '20 at 06:38
You should use `--packages` instead of downloading single jars — OneCricketeer, Jan 13 '20 at 06:38

score -2 · Answer 1 · answered Jan 13 '20 at 06:29

As per the documentation - You have to compile your streaming application into a JAR. If you are using spark-submit to start the application, then you will not need to provide Spark and Spark Streaming in the JAR. However, if your application uses advanced sources (e.g. Kafka, Flume), then you will have to package the extra artifact they link to, along with their dependencies, in the JAR that is used to deploy the application. For example, an application using KafkaUtils will have to include spark-streaming-kafka-0-10_2.12 and all its transitive dependencies in the application JAR.

Alternatively, you can use like "--packages org.apache.spark:spark-sql-kafka-0-10_2.12:2.4.0" with spark submit command.

The "streaming application" is already a JAR in this case. It is in the examples folder. The error isn't related to missing Kafka libraries, though — OneCricketeer, Jan 13 '20 at 06:31

NoSuchMethodError when running Spark streaming example with kafka

1 Answers1