0

I'm using spark-1.4.1, scala 2.10.5 and I code on eclipse IDE. I encounter the following error:

Error: Main() method not found 
Cannot export the JAR file as it does not recognize the main class

How to start with writing a spark code?

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
    def main(args: Array[String]) {
        val logFile = "YOUR_SPARK_HOME/README.md" // Should be some file on your system
        val conf = new SparkConf().setAppName("Simple Application")
        val sc = new SparkContext(conf)
        val logData = sc.textFile(logFile, 2).cache()
        val numAs = logData.filter(line => line.contains("a")).count()
        val numBs = logData.filter(line => line.contains("b")).count()
        println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
    }
}
Bacon
  • 1,814
  • 3
  • 21
  • 36
spark_dream
  • 326
  • 2
  • 8
  • 23
  • I changed "YOUR_SPARK_HOME" to where I installed spark – spark_dream Aug 14 '15 at 23:43
  • I guess you encounter that error at compile time, not while submitting your code, right? Are you sure that your classpath is actually including the spark assembly JAR? – Bacon Aug 15 '15 at 04:17
  • You don't need to actually declare a main object. Also, the OP will need to add a master to the sparkContext, for example `new SparkConf().setAppName("Simple Application").setMaster("local")` to set the master as local. – Bacon Aug 15 '15 at 12:47
  • Yes this is a compile time error. I could do solve that when I kept my directory structure correct. Thanks – spark_dream Aug 15 '15 at 16:32
  • After submitting the spark script I'm getting error: Exception in thread main():org.apache.hadoop.mapred.InvalidInputException hdfs://localhost:8020/.../src/main/scala/SimpleApp.scala does not exist. The input path I specified does not exist in hdfs . True because I specified it to be local. Why am I getting this error? @Bacon – spark_dream Aug 15 '15 at 16:57
  • My file /SampleApp is in the HDFS.Even then I'm getting this error – spark_dream Aug 15 '15 at 18:51
  • Please consider updating your question to detail how you solved it. Concerning your hdfs issue, you should give the file path relative to your hdfs root hierarchy, i.e check the result of `hdfs dfs -ls /` – Bacon Aug 15 '15 at 19:51
  • will try that thank you@Bacon – spark_dream Aug 16 '15 at 06:45
  • What if the file is not on hdfs ? which path should I give in the sc.textFile(" path?") – spark_dream Aug 16 '15 at 23:38

0 Answers0