SparkWordCount problem - java.lang.ClassNotFoundException

Question

This is not a duplicate question and i have tried so many ways to make this work, but did not work

I am trying to write a word count application so that i can run through spark-submit

I am using IntelliJ IDEA, spark - 2.1.1 and scala - 2.11.8

My word count code looks like this ::

package com.netflix.utilities

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

class SparkWordCount {
  def main(args: Array[String]) {
   // create Spark context with Spark configuration
   println("starting")
   val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

   // get threshold
   val threshold = args(1).toInt

   // read in text file and split each document into words
   val tokenized = sc.textFile(args(0)).flatMap(_.split(" "))

   // count the occurrence of each word
   val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)

   // filter out words with fewer than threshold occurrences
   val filtered = wordCounts.filter(_._2 >= threshold)

   // count characters
   val charCounts = filtered.flatMap(_._1.toCharArray).map((_, 1)).reduceByKey(_ + _)

   System.out.println(charCounts.collect().mkString(", "))
  }
}

my pom.xml looks like this ::

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.netflix.utilities</groupId>
<artifactId>SparkWordCount</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>

</dependencies>

<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Internal repo</name>
        <url>file:///Users/sankar.biswas/Noah/</url>
    </repository>
</distributionManagement>


<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

Now I have put my jar in the same folder i have my spark-submit

This is the spark-submit I am running ::

./spark-submit --class com.netflix.utilities.SparkWordCount --master local --deploy-mode client SparkWordCount-1.0-SNAPSHOT.jar "file:////Users/sankar.biswas/Desktop/hello.txt"

But I keep getting this error ::

java.lang.ClassNotFoundException: SparkWordCount

I am not getting what I am missing. Any suggestion would be highly appreciated!

You have to include package as well. Just a class name is not enough — Rishi Saraf, Jan 14 '19 at 08:25

score 0 · Answer 1 · answered Jan 14 '19 at 08:20

0

Instead of class SparkWordCount use object SparkWordCount and change --class "SparkWordCount" to --class com.netflix.utilities.SparkWordCount and run it

answered Jan 14 '19 at 08:20

Subhasish Guha

222
1
6

./spark-submit --class com.netflix.utilities.SparkWordCount --master local --deploy-mode client SparkWordCount-1.0-SNAPSHOT.jar "file:////Users/sankar.biswas/Desktop/hello.txt" 2 . ==> Getting the same error java.lang.ClassNotFoundException: com.netflix.utilities.SparkWordCount – Sankar Jan 14 '19 at 08:54
2

Have you changed it to Object? – Subhasish Guha Jan 14 '19 at 11:27
Change the class `SparkWordCount` to an `object`. – Afaq Jan 14 '19 at 12:55

SparkWordCount problem - java.lang.ClassNotFoundException

1 Answers1