Twitter Spark Streaming using Apache Spark 2.2.1 - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging

Question

Just started working on twitter spark streaming on eclipse IDE. I updated my pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>please.help</groupId>
    <artifactId>please</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>2.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>2.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-twitter_2.10</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-examples_2.10</artifactId>
            <version>1.0.2</version>
        </dependency>
    </dependencies>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
</project>

And I created a java class named TweetStream.java with the following codes:

public static void main(String[] args) {
    final String consumerKey = "****";
    final String consumerSecret = "****";
    final String accessToken = "****";
    final String accessTokenSecret = "****";

    SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("SparkTwitterHelloWorldExample");
    JavaStreamingContext jssc = new JavaStreamingContext(conf, new Duration(30000));

    System.setProperty("twitter4j.oauth.consumerKey", consumerKey);
    System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret);
    System.setProperty("twitter4j.oauth.accessToken", accessToken);
    System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret);

    JavaReceiverInputDStream<Status> twitterStream = TwitterUtils.createStream(jssc);

    // Without filter: Output text of all tweets
    JavaDStream<String> statuses = twitterStream.map(
            new Function<Status, String>() {
                public String call(Status status) { return status.getText(); }
            }
    );
    statuses.print();
    jssc.start();
}

Follwoing that, I did Run as > Maven install which built successfully. Then Run as > Run as Java application which gave the error:

    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/03/04 10:39:02 INFO SparkContext: Running Spark version 2.2.1
18/03/04 10:39:02 WARN SparkContext: Support for Scala 2.10 is deprecated as of Spark 2.1.0
18/03/04 10:39:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/04 10:39:03 INFO SparkContext: Submitted application: SparkTwitterHelloWorldExample
18/03/04 10:39:03 INFO SecurityManager: Changing view acls to: Xian
18/03/04 10:39:03 INFO SecurityManager: Changing modify acls to: Xian
18/03/04 10:39:03 INFO SecurityManager: Changing view acls groups to: 
18/03/04 10:39:03 INFO SecurityManager: Changing modify acls groups to: 
18/03/04 10:39:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(Xian); groups with view permissions: Set(); users  with modify permissions: Set(Xian); groups with modify permissions: Set()
18/03/04 10:39:04 INFO Utils: Successfully started service 'sparkDriver' on port 60829.
18/03/04 10:39:04 INFO SparkEnv: Registering MapOutputTracker
18/03/04 10:39:04 INFO SparkEnv: Registering BlockManagerMaster
18/03/04 10:39:04 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/03/04 10:39:04 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/03/04 10:39:04 INFO DiskBlockManager: Created local directory at C:\Users\User\AppData\Local\Temp\blockmgr-72db40d5-23de-4b88-98c1-30114f48828c
18/03/04 10:39:04 INFO MemoryStore: MemoryStore started with capacity 901.8 MB
18/03/04 10:39:04 INFO SparkEnv: Registering OutputCommitCoordinator
18/03/04 10:39:04 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/03/04 10:39:04 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.100.4:4040
18/03/04 10:39:04 INFO Executor: Starting executor ID driver on host localhost
18/03/04 10:39:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60842.
18/03/04 10:39:04 INFO NettyBlockTransferService: Server created on 192.168.100.4:60842
18/03/04 10:39:04 INFO BlockManager: Using     org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/03/04 10:39:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.100.4, 60842, None)
18/03/04 10:39:04 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.100.4:60842 with 901.8 MB RAM, BlockManagerId(driver, 192.168.100.4, 60842, None)
18/03/04 10:39:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.100.4, 60842, None)
18/03/04 10:39:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.100.4, 60842, None)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
        at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.streaming.twitter.TwitterUtils$.createStream(TwitterUtils.scala:44)
    at org.apache.spark.streaming.twitter.TwitterUtils$.createStream(TwitterUtils.scala:56)
    at org.apache.spark.streaming.twitter.TwitterUtils.createStream(TwitterUtils.scala)
    at please.TweetStream.main(TweetStream.java:27)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more
18/03/04 10:39:05 INFO SparkContext: Invoking stop() from shutdown hook
18/03/04 10:39:05 INFO SparkUI: Stopped Spark web UI at http://192.168.100.4:4040
18/03/04 10:39:05 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/03/04 10:39:05 INFO MemoryStore: MemoryStore cleared
18/03/04 10:39:05 INFO BlockManager: BlockManager stopped
18/03/04 10:39:05 INFO BlockManagerMaster: BlockManagerMaster stopped
18/03/04 10:39:05 INFO     OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/03/04 10:39:05 INFO SparkContext: Successfully stopped SparkContext
18/03/04 10:39:05 INFO ShutdownHookManager: Shutdown hook called
18/03/04 10:39:05 INFO ShutdownHookManager: Deleting directory C:\Users\User\AppData\Local\Temp\spark-b93e2698-f4fb-4d1a-b690-6f1128f4759c

The application started successfully, but a logging problem occurred and the application just skipped the codes and closed the connection.

Looks like Twitter Streaming library is not available for Spark 2 — OneCricketeer, Mar 04 '18 at 07:19
You certainly don't need the Spark examples dependency, but look at http://bahir.apache.org/docs/spark/current/spark-streaming-twitter/ — OneCricketeer, Mar 04 '18 at 07:21

Twitter Spark Streaming using Apache Spark 2.2.1 - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging

0 Answers0