Error with Spark: NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less

Question

I've got an application which streams Twitter posts. It worked correctly, but after modifying the build path, I didn't manage to set it how it was before. I keep getting this error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getSystemProperties(Utils.scala:1710)
at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:73)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:68)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:55)
at Analytics.init(Analytics.java:20)
at KafkaTwitterProducer.main(KafkaTwitterProducer.java:162)

This is my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>TwitterStreamingAnalyzer</groupId>
  <artifactId>TwitterStreamingAnalyzer</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
   <dependency>  
                <groupId>org.apache.kafka</groupId>  
                <artifactId>kafka_2.10</artifactId>  
                <version>0.10.2.1</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-core -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-core</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-stream</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-async -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-async</artifactId>  
                <version>4.0.4</version>  
           </dependency> 
           
     </dependencies> 
</project>

I already read all of the similar questions, but none of them helped me, because I tried to import the right versions of the jars, but it still doesn't work. How can I fix this?

You should have conflicting libraries in your projects dependency. Two versions of scala library. You will have to delete one of them — Ramesh Maharjan, Jun 05 '17 at 16:30

score 2 · Accepted Answer · answered Jun 05 '17 at 17:01

Looks like you are trying to use a kafka library compiled in Scala 2.10 in a Spark cluster compiled in other version (probably 2.11). Try this change in your pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>TwitterStreamingAnalyzer</groupId>
  <artifactId>TwitterStreamingAnalyzer</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
   <dependency>  
                <groupId>org.apache.kafka</groupId>  
                <artifactId>kafka_2.11</artifactId>  
                <version>0.10.2.1</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-core -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-core</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-stream</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-async -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-async</artifactId>  
                <version>4.0.4</version>  
           </dependency> 

     </dependencies> 
</project>

Hi @Mikel, I've got this problem again. This time pasting your edited pom.xml is not enough. Any suggestion? — sirdan, Jun 14 '17 at 15:33
did you change the version where you are executing Spark? for example if you are using EMR, did you change the version? — Mikel San Vicente, Jun 14 '17 at 16:44
I cleaned up the build path, I tried to reimport the same jars but there must be something wrong. I think it's just a matter of jars, because I haven't changed anything. — sirdan, Jun 14 '17 at 16:46
where are you submitting the spark app? EMR cluster? cloudera? did that change? or are you executing this from your IDE? — Mikel San Vicente, Jun 14 '17 at 16:49
are you executing using an IDE? if yes, what IDE? IntelliJ? you might want to check the scala version selected for your project in that IDE — Mikel San Vicente, Jun 14 '17 at 16:59
I don't use eclipse but this post might help https://stackoverflow.com/questions/10847049/how-can-i-find-the-version-of-scala-installed-in-eclipse-ide — Mikel San Vicente, Jun 14 '17 at 17:01

Error with Spark: NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less

1 Answers1

Linked