I am trying to load csv file to dataframe by using spark-csv_2.10-1.3.0
sqlContext.read.format('com.databricks.spark.csv')
.options(header='true',dateFormat='dd/MM/YYYY hh:mm')
.load('test.csv',schema = s_acr_rv_derivative_schema)
the schema:
s_acr_rv_derivative_schema = StructType([
StructField("COUNTERPARTY", StringType(), True),
StructField("ORIG_COUNTERPARTY", StringType(), True),
StructField("CUSTOMER_NAME", StringType(), True),
StructField("TRADE_ID", StringType(), True),
StructField("TRADE_DATE", DateType(), True)]
Issue:
Caused by: java.lang.IllegalArgumentException
at java.sql.Date.valueOf(Date.java:140)
at com.databricks.spark.csv.util.TypeCast$.castTo(TypeCast.scala:64)