Scala Spark streaming kafka

Question

I have created a sample topic in kafka and i am trying to consume the content in spark using below script:

import org.apache.spark._
 import org.apache.spark.streaming._
 import org.apache.spark.streaming.kafka._
 import org.apache.kafka.common.serialization.StringDeserializer
 import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import 
org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe


 class Kafkaconsumer {
  val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "host1:port,host2:port2,host3:port3",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "use_a_separate_group_id_for_each_stream",
  "auto.offset.reset" -> "latest",
  "enable.auto.commit" -> (false: java.lang.Boolean)
  )
  val sparkConf = new SparkConf().setMaster("yarn")
 .setAppName("kafka example")
  val streamingContext = new StreamingContext(sparkConf, Seconds(10))
  val topics = Array("topicname")
  val topicsSet = topics.split(",").toSet
  val stream = KafkaUtils.createDirectStream[String, String](
  streamingContext,
  PreferConsistent,
  Subscribe[String, String](kafkaParams,topicsSet)
  )
  stream.print()
  stream.map(record => (record.key, record.value))
  streamingContext.start()
  streamingContext.awaitTermination()

I also have included the necessary libraries for executing the code.

I have the below error, kindly let me know how to solve this problem.

Error:
 Error:(23, 27) wrong number of type parameters for overloaded method value createDirectStream with alternatives:
  [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V]](jssc: org.apache.spark.streaming.api.java.JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], kafkaParams: java.util.Map[String,String], topics: java.util.Set[String])org.apache.spark.streaming.api.java.JavaPairInputDStream[K,V] <and>
  [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V], R](jssc: org.apache.spark.streaming.api.java.JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], recordClass: Class[R], kafkaParams: java.util.Map[String,String], fromOffsets: java.util.Map[kafka.common.TopicAndPartition,Long], messageHandler: org.apache.spark.api.java.function.Function[kafka.message.MessageAndMetadata[K,V],R])org.apache.spark.streaming.api.java.JavaInputDStream[R] <and>
  [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V]](ssc: org.apache.spark.streaming.StreamingContext, kafkaParams: Map[String,String], topics: Set[String])(implicit evidence$19: scala.reflect.ClassTag[K], implicit evidence$20: scala.reflect.ClassTag[V], implicit evidence$21: scala.reflect.ClassTag[KD], implicit evidence$22: scala.reflect.ClassTag[VD])org.apache.spark.streaming.dstream.InputDStream[(K, V)] <and>
  [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V], R](ssc: org.apache.spark.streaming.StreamingContext, kafkaParams: Map[String,String], fromOffsets: Map[kafka.common.TopicAndPartition,Long], messageHandler: kafka.message.MessageAndMetadata[K,V] => R)(implicit evidence$14: scala.reflect.ClassTag[K], implicit evidence$15: scala.reflect.ClassTag[V], implicit evidence$16: scala.reflect.ClassTag[KD], implicit evidence$17: scala.reflect.ClassTag[VD], implicit evidence$18: scala.reflect.ClassTag[R])org.apache.spark.streaming.dstream.InputDStream[R]val stream = KafkaUtils.createDirectStream[String, String](

Could you please post the entire error trace? Also, are these 2 statements really working? val topics = Array("topicname") val topicsSet = topics.split(",").toSet - because it seems like you're trying to split an array on the comma symbol. You should probably get an error "split is not a member of Array[String]". — Lalit, Sep 15 '18 at 12:00
Thanks for the response as you said i am getting the same error. — Aish Mahesh, Sep 16 '18 at 15:05
I added that for some other reason sorry and now i removed that line and having the same code with below error: Error:(25, 5) type mismatch; found : org.apache.spark.streaming.kafka010.LocationStrategy required: Map[String,String] PreferConsistent, — Aish Mahesh, Sep 16 '18 at 15:06
There is always an error in the createDirectStream syntax,when i add KafkaUtils.createDirectStream[String, String] then Error: wrong number of type parameters for overloaded method value createDirectStream with alternatives. — Aish Mahesh, Sep 16 '18 at 15:15
Could you please edit the post and share the details of your various attempts? as you've mentioned two different issues in your attempts i.e. "Wrong number of type parameters" and "type mismatch". I'm actually trying to reproduce this issue in my system and will then attempt a fix. — Lalit, Sep 16 '18 at 15:32

score 0 · Answer 1 · answered Sep 16 '18 at 16:05

0

Add type parametres for the type of decoding of the key and value you need, for example:

Change:

KafkaUtils.createDirectStream[String, String](
  streamingContext,
  PreferConsistent,
  Subscribe[String, String](kafkaParams,topicsSet)
)

to

KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
  streamingContext,
  PreferConsistent,
  Subscribe[String, String](kafkaParams,topicsSet)
)

answered Sep 16 '18 at 16:05

bp2010

2,342
17
34

Thanks a lot for the response i did try the above option that you mentioned and it says unable to resolve symbol stringdecoder.I am searching many ways to solve the problem if you get any other solution do share as i am stuck at this point. – Aish Mahesh Sep 17 '18 at 14:46
which versions of spark, spark-streaming, and spark-streaming-kafka are you using, please share these – bp2010 Sep 17 '18 at 14:47
sparkVersion = "1.6.3",scalaVersion := "2.12.6",spark-streaming_2.10 – Aish Mahesh Sep 17 '18 at 15:47
Also if you could provide me an insight on how to build fat jars from this it would be really helpful for me – Aish Mahesh Sep 17 '18 at 18:27
if you are using `spark-streaming_2.10 `, then the scala version should also be `scalaVersion := "2.10.x ` – bp2010 Sep 17 '18 at 19:43
and what version of spark-streaming-kafka ?? Can you post your configuration.. pom.xml or build.sbt whichever you use – bp2010 Sep 17 '18 at 19:44
Build.sbt:import sbt._ import Keys._ name := "consumerspark.scala" version := "0.1" scalaVersion := "2.12.6" val sparkVersion = "1.6.3" libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3" libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0" – Aish Mahesh Sep 17 '18 at 20:57
Plugin.sbt:addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6") resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns) – Aish Mahesh Sep 17 '18 at 20:57
the issue is that you are using different spark versions (1.6.3 and 2.0.0).. check this answer to have the correct versions in build.sbt: https://stackoverflow.com/a/52355875/3213111 – bp2010 Sep 18 '18 at 07:38

Scala Spark streaming kafka

1 Answers1