0

In spark data set I'm using createTableColumnTypes for the database column data types to use instead of the defaults, when creating the table. It's working perfect for VARCHAR(n) but if I'm using TEXT it's throwing error. Code is written in Java

Dataset < Row > df = sqlContext.read()
                    .format("com.databricks.spark.csv")
                    .schema(sparkDataSchema)
                    .option("delimiter", dataDelimeter)
                    .option("header", true).option("quote", "\"")
                    .option("escape", "\"").load(paths.split(","));


df.write().option("truncate", true).mode(SaveMode.Overwrite).option("createTableColumnTypes", customSchema).jdbc(url, tableName, connectionProperties);

Error is given below

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: DataType text is not supported.(line 1, pos 29)

blackbishop
  • 30,945
  • 11
  • 55
  • 76
Dev
  • 413
  • 10
  • 27
  • share the custom schema that you have specified. – dassum Dec 23 '19 at 16:45
  • @dassum custom schema is id long, name VARCHAR(150), comment TEXT – Dev Dec 24 '19 at 06:51
  • 1
    Please use the StructField("comment", DataTypes.StringType, true, Metadata.empty()) for comment column. TEXT type not supported by Spark. – dassum Dec 24 '19 at 07:02
  • @dassum yes I know I can use in StructField but I was finding another way to do through createTableColumnTypes option, anyway thanks for your input. – Dev Dec 24 '19 at 09:53

0 Answers0