2

I have a cluster with Spark 2.0 and Zeppelin 0.6.1 installed. Since the class TwitterUtils.scala is moved from Spark project to Apache Bahir, I can't use the TwitterUtils in my Zeppelin notebook anymore.

Here the snippets of my notebook:

Dependency loading:

%dep
z.reset
z.load("org.apache.bahir:spark-streaming-twitter_2.11:2.0.0")

DepInterpreter(%dep) deprecated. Remove dependencies and repositories through GUI interpreter menu instead.
DepInterpreter(%dep) deprecated. Load dependency through GUI interpreter menu instead.
res1: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Dependency@4793109a

And the Spark part:

import org.apache.spark.streaming.twitter
import org.apache.spark.streaming._
import org.apache.spark.storage.StorageLevel
import scala.io.Source
import scala.collection.mutable.HashMap
import java.io.File
import org.apache.log4j.Logger
import org.apache.log4j.Level
import sys.process.stringSeqToProcess
import org.apache.spark.SparkConf

// ********************************* Configures the Oauth Credentials for accessing Twitter ****************************
def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {...}

// ***************************************** Configure Twitter credentials ********************************************
val apiKey = ...
val apiSecret = ...
val accessToken = ...
val accessTokenSecret = ...
configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)

//  ************************************************* The logic itself *************************************************
val ssc = new StreamingContext(sc, Seconds(2))
val tweets = TwitterUtils.createStream(ssc, None)
val twt = tweets.window(Seconds(60))

When I try to run the Spark part of the notebook after importing the dependency, I get the following exception:

<console>:44: error: object twitter is not a member of package org.apache.spark.streaming
   import org.apache.spark.streaming.twitter

What am I doing wrong here? Bahir documentation also uses the import org.apache.spark.streaming.twitter._ command, see http://bahir.apache.org/docs/spark/2.0.0/spark-streaming-twitter/

D. Müller
  • 3,336
  • 4
  • 36
  • 84

1 Answers1

1

Well, dep is not exactly stable and since it is deprecated anyway why not use supported methods? If you don't won't to modify neither Spark nor Zeppelin configuration files you can add dependencies to the interpreter configuration (I omitted properties for clarity):

enter image description here

zero323
  • 322,348
  • 103
  • 959
  • 935
  • Ah yea, that did the trick. I haven't found the dependencies section in the Spark interpreter yet! It works perfectly now, thank you! – D. Müller Sep 12 '16 at 14:12
  • What do you mean by 'If you don't won't to modify neither Spark nor Zeppelin configuration files' ? Where exactly can i find these configuration files, to modify for adding these dependencies – Saurabh Mishra Feb 06 '17 at 09:25
  • Maybe this site is helpful for you, it describes how to add dependencies via the zeppelin web UI: https://zeppelin.apache.org/docs/0.6.1/manual/dependencymanagement.html See "Load Dependencies to Interpreter" section – D. Müller Feb 06 '17 at 10:27