Spark CSV Escape Not Working

Question

I am using spark-core version 2.0.1 with Scala 2.11. I have simple code to read a csv file which has \ escapes.

val myDA = spark.read
      .option("quote",null)
    .schema(mySchema)
    .csv(filePath)

As per documentation \ is default escape for csv reader. But it does not work. Spark is reading \ as part of my data. For Ex: City column in csv file is north rocks\,au . I am expecting city column should read in code as northrocks,au. But instead spark reads it as northrocks\ and moves au to next column.

I have tried following but did not work:

Explicitly defined escape .option("escape","\")
Changed escape to | or : in file and in code
I have tried using spark-csv library

Any one facing same issue? Am I missing something?

Thanks

can replace the way you separate between these? lie `north rocks:au`. I would also try wrapping the whole thing in quotes like `'north rocks\,au'` — raam86, Oct 27 '16 at 15:19
This could be the cause: https://issues.apache.org/jira/browse/CSV-135 — JNish, Oct 27 '16 at 16:00
Interesting finding: Escaping works if data is quoted but not otherwise. — JNish, Oct 27 '16 at 17:25

score 0 · Answer 1 · answered Oct 27 '16 at 21:59

0

So escape will only work with quotes. Recommendation is to use quotes if you want to escape.

https://github.com/databricks/spark-csv/issues/390

answered Oct 27 '16 at 21:59

JNish

145
2
10

Spark CSV Escape Not Working

1 Answers1