Any examples on using spark_read_csv correctly?

Question

when I run this:

terrorism <- spark_read_csv(sc, name = "terrorism", path = “\home\osboxes\Desktop\globalterrorismdb_0718dist.csv”)

I get this error:

Error: unexpected input in "terrorism <- spark_read_csv(sc, name = "terrorism", path = �".

Please any ideas to debug this?

is this mac or windows? either way, check your path name, and maybe try using `/` (forward slash) not `\` (backslash) — morgan121, Mar 03 '20 at 22:39
why are you getting this symbol �". ?? in the error message? try to read the file using the read file api in R, and print few lines . then use read_csv — Remis Haroon - رامز, Mar 04 '20 at 08:56

score 0 · Answer 1 · answered Mar 04 '20 at 13:00

Your second pair of quotes looks weird; first pair looks fine. This si the standard way of redaing a CSV in the Spark world.

val peopleDFCsv = spark.read.format("csv")
  .option("sep", ";")
  .option("inferSchema", "true")
  .option("header", "true")
  .load("examples/src/main/resources/people.csv")

Any examples on using spark_read_csv correctly?

1 Answers1