Spark Streaming + kafka "JobGenerator" java.lang.NoSuchMethodError

Question

I'm new in spark streaming and kafka and I don't understand this runtime exception. I've already setup the kafka server.

    Exception in thread "JobGenerator" java.lang.NoSuchMethodError: org.apache.spark.streaming.scheduler.InputInfoTracker.reportInfo(Lorg/apache/spark/streaming/Time;Lorg/apache/spark/streaming/scheduler/StreamInputInfo;)V
at org.apache.spark.streaming.kafka.DirectKafkaInputDStream.compute(DirectKafkaInputDStream.scala:166)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:350)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:349)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:344)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:342)
at scala.Option.orElse(Option.scala:257)

and this is my code

public class TwitterStreaming {
     // setup kafka : 
     public static final String ZKQuorum = "localhost:2181";
     public static final String ConsumerGroupID = "ingi2145-analytics";
     public static final String ListTopics = "newTweet";
     public static final String ListBrokers = "localhost:9092"; // I'm not sure about ...

    @SuppressWarnings("deprecation")
public static void main(String[] args) throws Exception {
    // Location of the Spark directory
    String sparkHome = "usr/local/spark";
    // URL of the Spark cluster
    String sparkUrl = "local[4]";
    // Location of the required JAR files
    String jarFile = "target/analytics-1.0.jar";
// Generating spark's streaming context
JavaStreamingContext jssc = new JavaStreamingContext(
  sparkUrl, "Streaming", new Duration(1000), sparkHome, new String[]{jarFile});
// Start kafka stream
HashSet<String> topicsSet = new HashSet<String>(Arrays.asList(ListTopics.split(",")));
HashMap<String, String> kafkaParams = new HashMap<String, String>();
kafkaParams.put("metadata.broker.list", ListBrokers);

//JavaPairReceiverInputDStream<String, String> kafkaStream = KafkaUtils.createStream(ssc, ZKQuorum, ConsumerGroupID, mapPartitionsPerTopics);
// Create direct kafka stream with brokers and topics
JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
    jssc,
    String.class,
    String.class,
    StringDecoder.class,
    StringDecoder.class,
    kafkaParams,
    topicsSet
);

// get the json file :
   JavaDStream<String> json = messages.map(
        new Function<Tuple2<String, String>, String>() {
            public String call(Tuple2<String, String> tuple2) {
              return tuple2._2();
            }
   });

The aim of this project is to compute the 10 bests hashtags from a twitter stream by using a kafka queue. The code was working without kakfa. Have you got an idea of what's the problem ?

Answer 1

I had the same issue and it was the version of spark I was using. I was using 1.5, then used 1.4 and ultimately the version that worked for me was 1.6. So, please make sure that the Kafka version you are using is compatible with with the Spark version. In my case, I'm using Kafka version 2.10-0.10.1.1 with spark-1.6.0-bin-hadoop2.3.

Also, (very important) make sure you are not getting any forbidden error in your log files. You have to assign the proper security grants to folders used by spark, otherwise you may receive a lot of errors that has nothing to do with the application itself but with improper security setup.