AbstractMethodError upon creation of new StreamingContext

Question

I've been having problems trying to instantiante a new StreamingContext of Spark Streaming.

I'm trying to create a new StreamingContext, but an error of AbstractMethodError is being thrown. I've been debugging the stack trace and found out that when the third Spark ListenerBus is being created inside the StreamingListenerBus, the application stops and throws this error.

Below is the code I'm trying to execute

package platform.etl

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

object ClickGeneratorStreaming {
  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("ClickGeneratorStreaming").setMaster("local[*]")
    val ssc = new StreamingContext(conf, Seconds(10)

  }
}

And here is the stack trace

Exception in thread "main" java.lang.AbstractMethodError
    at org.apache.spark.util.ListenerBus$class.$init$(ListenerBus.scala:35)
    at org.apache.spark.streaming.scheduler.StreamingListenerBus.<init>(StreamingListenerBus.scala:30)
    at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:56)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:183)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
    at platform.etl.ClickGeneratorStreaming$.main(ClickGeneratorStreaming.scala:10)
    at platform.etl.ClickGeneratorStreaming.main(ClickGeneratorStreaming.scala)

My build.sbt

name := "spark"

version := "0.1"

scalaVersion := "2.11.0"

val sparkVersion = "2.3.0.2.6.5.0-292"
val sparkKafkaVersion = "2.3.0"
val argonautVersion = "6.2"

resolvers += "jitpack" at "https://jitpack.io"
resolvers += "horton" at "http://repo.hortonworks.com/content/repositories/releases"
resolvers += "horton2" at "http://repo.hortonworks.com/content/groups/public"


libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.7.3.2.6.5.0-292" excludeAll ExclusionRule(organization = "javax.servlet")
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.4"
libraryDependencies += "com.softwaremill.sttp" %% "core" % "1.2.0-RC2"
libraryDependencies += "com.softwaremill.retry" %% "retry" % "0.3.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "com.github.scopt" %% "scopt" % "3.7.0"
libraryDependencies += "io.argonaut" %% "argonaut" % argonautVersion
libraryDependencies += "io.argonaut" %% "argonaut-monocle" % argonautVersion
libraryDependencies += "com.github.scopt" %% "scopt" % "3.7.0"
libraryDependencies += "com.github.mrpowers" % "spark-fast-tests" % "v2.3.0_0.11.0" % "test"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.5"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.3.0"
libraryDependencies += "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.2.2"
libraryDependencies += "com.redislabs" % "spark-redis" % "2.3.1-M2"
libraryDependencies +=  "org.scalaj" %% "scalaj-http" % "2.4.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion 
libraryDependencies += "org.apache.spark" %% "spark-hive" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % sparkVersion


assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "application.conf"            => MergeStrategy.concat
  case "reference.conf"              => MergeStrategy.concat
  case _ => MergeStrategy.first
}

My plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

Could you also include plugins.sbt please? Is it only assembly plugin you use? — Mon Calamari, Aug 01 '19 at 17:40
Classic dependency hell. Do you really need that many dependencies? Can you not start with latest spark streaming? — Mon Calamari, Aug 01 '19 at 18:09
I've included my plugins.sbt, this is a pretty big project, unfortunately, and one part of it must have a real-time processing pipeline. — Luan Araldi, Aug 01 '19 at 18:21
While trying to find out more information about the problems, I've removed my ivy cache and redownloaded the dependencies, and this message came up [warn] * org.apache.spark:spark-tags_2.11:2.3.0.2.6.5.0-292 is selected over 2.1.0 [warn] +- org.apache.spark:spark-streaming_2.11:2.1.0 (depends on 2.1.0) — Luan Araldi, Aug 01 '19 at 18:22
There is a lot of evictions in your project. Try running https://github.com/jrudolph/sbt-dependency-graph to find out offensive libraries. — Mon Calamari, Aug 01 '19 at 19:02

score 0 · Accepted Answer · answered Aug 01 '19 at 19:52

Found the problem. Looks like I forgot to add the spark-streaming dependency on my build.sbt, and for some reason, It found a way to use a dependency of dependency on my import, making it use a different version of spark-streaming which is not compatible with my spark version.

To solve that I just added a single newline at my build.sbt

libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion

And it works flawlessly now.

AbstractMethodError upon creation of new StreamingContext

1 Answers1