How to load a local file with a local spark session

Asked Apr 16 '18 at 19:26

Active Jan 18 '19 at 03:56

Viewed 2,374 times

I'm running a local spark session on my mac via my intellij sbt console and I get a org.apache.spark.sql.AnalysisException: Path does not exist: file:/Users/myuser/Documents/data/dataset.csv; error.

my current code looks like this:
val data = spark.read.csv("file:///Users/myuser/Documents/data/dataset.csv")

I've also tried:

val data = spark.read.csv("/Users/myuser/Documents/data/dataset.csv")

my spark session looks like this

import org.apache.spark.sql.SparkSession

trait SparkSessionWrapper {

  lazy val spark: SparkSession = {
    SparkSession
      .builder()
      .master("local")
      .appName("avro_test")
      .getOrCreate()
  }

}

I know this is the same issue as the one found here: How to load local file in sc.textFile, instead of HDFS

but none of the answers here (and others i've looked at) are helping me or else i'm not fully understanding them. any suggestions?

asked Apr 16 '18 at 19:26

J.Hammond

also i'm using spark 2.2.0 – J.Hammond Apr 16 '18 at 19:27
1

Nothing wrong with the `SparkSession`. It has been created correctly. Are you sure that path you had mentioned indeed exists in your machine? – Sivaprasanna Sethuraman Apr 17 '18 at 01:46
Your right! there was a error in my path and that was it, How embarrassing. thank you – J.Hammond Apr 17 '18 at 13:32

How to load a local file with a local spark session

0 Answers0