Spark CSV reader : garbled Japanese text and handling multilines

Question

In my Spark job (spark 2.4.1) , I am reading CSV files on S3.These files contain Japanese characters.Also they can have ^M character (u000D) so I need to parse them as multiline.

First I used following code to read CSV files:

  implicit class DataFrameReadImplicits (dataFrameReader: DataFrameReader) {
     def readTeradataCSV(schema: StructType, s3Path: String) : DataFrame = {

        dataFrameReader.option("delimiter", "\u0001")
          .option("header", "false")
          .option("inferSchema", "false")
          .option("multiLine","true")
          .option("encoding", "UTF-8")
          .option("charset", "UTF-8")
          .schema(schema)
          .csv(s3Path)
     }
  }

But when I read DF using this method all the Japanese characters are garbled.

After doing some tests I found out that If I read the same S3 file using "spark.sparkContext.textFile(path)" Japanese characters encoded properly.

So I tried this way :

implicit class SparkSessionImplicits (spark : SparkSession) {
    def readTeradataCSV(schema: StructType, s3Path: String) = {
      import spark.sqlContext.implicits._
      spark.read.option("delimiter", "\u0001")
        .option("header", "false")
        .option("inferSchema", "false")
        .option("multiLine","true")
        .schema(schema)
        .csv(spark.sparkContext.textFile(s3Path).map(str => str.replaceAll("\u000D"," ")).toDS())
    }
  }

Now the encoding issue is fixed.However multilines doesn't work properly and lines are broken near ^M character , even though I tried to replace ^M using str.replaceAll("\u000D"," ")

Any tips on how to read Japanese characters using first method, or handle multi-lines using the second method ?

UPDATE: This encoding issue happens when the app runs on the Spark cluster.When I ran the app locally, reading the same S3 file, encoding works just fine.

Spark must read & parse the CSV into a DataFrame _before_ you can apply a Map operator to that DataFrame -- the damage is done. — Samson Scharfrichter, May 19 '20 at 08:21

score 1 · Answer 1 · answered May 19 '20 at 09:01

Some things are in the code but not (yet) in the docs. Did you try setting explicitly your line separator, thus avoiding the "multiline" workaround because of ^M?

From the unit tests for Spark "TextSuite" branch 2.4
https://github.com/apache/spark/blob/branch-2.4/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala

def testLineSeparator(lineSep: String): Unit = {
  test(s"SPARK-23577: Support line separator - lineSep: '$lineSep'") {
  ...
}
// scalastyle:off nonascii
Seq("|", "^", "::", "!!!@3", 0x1E.toChar.toString, "아").foreach { lineSep =>
  testLineSeparator(lineSep)
}
// scalastyle:on nonascii

From the source code for CSV options parsing, branch 3.0
https://github.com/apache/spark/blob/branch-3.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala

val lineSeparator: Option[String] = parameters.get("lineSep").map { sep =>
  require(sep.nonEmpty, "'lineSep' cannot be an empty string.")
  require(sep.length == 1, "'lineSep' can contain only 1 character.")
  sep
}
val lineSeparatorInRead: Option[Array[Byte]] = lineSeparator.map { lineSep =>
  lineSep.getBytes(charset)
}

So, looks like CSV does not support strings for line delimiters, just single characters, because it relies on some Hadoop library. I hope that's fine in your case.

The matching JIRAs are...

SPARK-21289 Text based formats do not support custom end-of-line delimiters ...
SPARK-23577 specific to text datasource > fixed in V2.4.0

score 0 · Answer 2 · answered May 18 '20 at 13:21

0

if your data is enclosed by double quote then you can use escape property.

df = (spark.read
 .option("header", "false")
 .csv("******",multiLine=True, escape='"')
)

answered May 18 '20 at 13:21

S-5618

11
2

1

thanks.but this is related to encoding not escaping – Ashika Umanga Umagiliya May 19 '20 at 04:51

Spark CSV reader : garbled Japanese text and handling multilines

2 Answers2

Linked