2

During migration from spark 1.6.2 to spark 2.0.0 appeared that package org.apache.spark.streaming.twitter has been removed and twitter streaming is no longer available as well as dependency

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-twitter_2.11</artifactId>
  <version>2.0.0</version>
</dependency>

Can anyone suggest how to procced twitter stream in new spark?

Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
Ivan Shulak
  • 100
  • 6

1 Answers1

7

Twitter (and some other) driver support has been removed in Spark 2.0.

You can see it in the removal section of the Release Notes:

Removals

The following features have been removed in Spark 2.0:

  • Less frequently used streaming connectors, including Twitter, Akka, MQTT, ZeroMQ

They have been extracted as a separate package under the Bahir Project. The twitter extension, streaming-twitter, can be found via:

sbt:

libraryDependencies += "org.apache.bahir" %% "spark-streaming-twitter" % "2.0.0"

Maven:

<dependency>
  <groupId>org.apache.bahir</groupId>
  <artifactId>spark-streaming-twitter_2.11</artifactId>
  <version>2.0.0-preview</version>
</dependency>

More on that (thanks to @IvanShulak) in the Mailing List

Edit:

For Spark 2.0.1, use:

libraryDependencies += "org.apache.bahir" %% "spark-streaming-twitter" % "2.0.1"
Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
  • thanks Yuval for replay. Is it dependency available in global maven repository? Cause I can't find it in [maven repo](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.spark-project%22) and also getting error in my pom file: Missing artifact org.spark-project:dstream-twitter_2.10:jar:0.1.0 – Ivan Shulak Aug 02 '16 at 10:05
  • @IvanShulak I guess it should be available soon. There's an open [bug about](https://github.com/spark-packages/dstream-twitter/issues/1) it – Yuval Itzchakov Aug 02 '16 at 10:16
  • thanks for help. I've updated your answer and added correct mvn dependency following your link in previous comment. – Ivan Shulak Aug 04 '16 at 07:54
  • @IvanShulak Thanks for the update Ivan. Modified the original answer to suit the findings. – Yuval Itzchakov Aug 04 '16 at 07:54
  • You should now be able to use: "org.apache.bahir" % "spark-streaming-twitter_2.11" % "2.0.1" – azuras Nov 15 '16 at 06:38