1

Using spark 1.6 I tried following code:

val diamonds = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/got_own/com_sep_fil.csv")

which caused the error

error: not found: value spark
Shaido
  • 27,497
  • 23
  • 70
  • 73
abdul sattar
  • 11
  • 1
  • 2
  • You need to define a `SparkContext` (in Spark 2.0+, a `SparkSession` can be used), see e.g. here: https://spark.apache.org/docs/1.6.1/quick-start.html#self-contained-applications – Shaido Apr 29 '20 at 08:03

3 Answers3

0

In Spark 1.6 shell you get sc of type SparkContext, not spark of type SparkSession, if you want to get that functionlity you will need to instantiate a SqlContext

import org.apache.spark.sql._
val spark = new SQLContext(sc)
Mikel San Vicente
  • 3,831
  • 2
  • 21
  • 39
  • Dude literally i am beginner in spark ,can you tell some more about query ,i didn't get you – abdul sattar Apr 29 '20 at 11:23
  • you are trying to use a variable called `spark` in Spark shell 1.6 but this variable was introduced in Spark shell 2.0 so the command fails because `spark` does not exist. In Spark 1.6 shell you can create a val `spark` by executing the code that I added in my post. – Mikel San Vicente Apr 29 '20 at 13:45
0

sqlContext is implict object SQL contect which can be used to load csv file and use com.databricks.spark.csv for mentionin csv file format

val df = sqlContext.read.format("csv").option("header", "true").option("inferSchema", "true").load("data.csv")
QuickSilver
  • 3,915
  • 2
  • 13
  • 29
  • .ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) – abdul sattar May 01 '20 at 06:16
  • updated the ans to read csv and not "com.databricks.spark.csv" – QuickSilver May 01 '20 at 06:22
0

You need to initialize instance using SQLContext(spark version<2.0) or SparkSession(spark version>=2.0) to use methods provided by Spark.

To initialize spark instance for spark version < 2.0 use:

import org.apache.spark.sql._
val spark = new SQLContext(sc)

To initialize spark instance for spark version >= 2.0 use:

val spark = new SparkConf().setAppName("SparkSessionExample").setMaster("local")

To read the csv using spark 1.6 and databricks spark-csv package: val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("data.csv")

Shrey Jakhmola
  • 522
  • 3
  • 9
  • dude spark told me for package , so problem is that i don't know what package i have to install in 1.6 ,i have tried this : $SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0 this is showing the error : error: ';' expected but '.' found. which is sossimple to under stand but don't know why it is showing me this – abdul sattar May 01 '20 at 06:45
  • You need to install the spark-csv package. Refer the link . Make sure you are using the proper version of Scala. If your spark is compiled using scala 2.10 then use ```$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.5.0``` and if scala 2.11 then use ```$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0```. – Shrey Jakhmola May 02 '20 at 08:15
  • i am using scala 2.10.5 ,when i paste your above package then it was showing me the error like : error: ';' expected but '.' found. which is confusing me bro – abdul sattar May 11 '20 at 10:43
  • @abdulsattar provide your ```spark-shell``` or ```spark-submit``` statement you are using – Shrey Jakhmola May 12 '20 at 12:07
  • bro i didn't get you? – abdul sattar May 15 '20 at 19:23