0

Having a very frustrating time trying trying to get spark not to split my column when it has a \ in the column value.

what do I need to adjust to do this. Not getting it

thank you :)

datapath = "abfss:/location"

df = (spark
      .read
      .option("header","true")
      .option("inferSchema","true")
      .option("quote","\"")
      .option("escape","'")
      .option("sep",",")
      .csv(datapath))
df.display()
Koedlt
  • 4,286
  • 8
  • 15
  • 33
Vanessa van gelder
  • 13
  • 2
  • Can you share sample values that are splitting incorrectly right now?What you are specifying right now means if value for any column is enclosed in double quotes, it will allow commas in that value. If quotes are to appear in values, they must be preceded by a single quote. – xenodevil Aug 24 '23 at 09:21

0 Answers0