what is the alternative for double datatype from spark sql(Databricks) to Sql Server Data warehouse

Question

I have to load the data from azure datalake to data warehouse.I have created set up for creating external tables.there is one column which is double datatype, i have used decimal type in sql server data warehouse for creating the external table and file format is parquet.But using csv it is working. i'm getting the following error.

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: ClassCastException: class java.lang.Double cannot be cast to class parquet.io.api.Binary (java.lang.Double is in module java.base of loader 'bootstrap'; parquet.io.api.Binary is in unnamed module of loader 'app'.

Can some one help me on this issue? Thanks in advance.

CREATE EXTERNAL TABLE [dbo].[EXT_TEST1]
( A VARCHAR(10),B decimal(36,19))) 
(DATA_SOURCE = [Azure_Datalake],LOCATION = N'/A/B/PARQUET/*.parquet/',FILE_FORMAT =parquetfileformat,REJECT_TYPE = VALUE,REJECT_VALUE = 1)

Column datatype in databricks:
A string,B double 
Data: A  |  B
      'a'  100.0050

Have you tried float? I think it will be considered a floating point therefore decimal will not work. — simon_dmorias, Apr 23 '19 at 12:50
@simon- Thanks for your reply. i have tried float it is working but i want an alternate way for double type in DWH — pythonUser, Apr 24 '19 at 10:59

score 0 · Answer 1 · answered Jul 12 '20 at 21:16

0

Use float(53) which is of 53 digits precision and 8 bytes length.

answered Jul 12 '20 at 21:16

Prags

1

what is the alternative for double datatype from spark sql(Databricks) to Sql Server Data warehouse

1 Answers1