0

The project I am working receive data in the form a Avro files. I am writing a generic code in Scala (using Scala IDE) to read all Avro files present in a folder and create a table for each of the avro file.

I am reading the avro file data as a dataframe and using below code to write the same back to Impala database (using JDBC).

df.write.mode("overwrite").format("jdbc").options(jdbcOptions).saveAsTable("table1")

When I execute this code, I receive an error with create statement.

[Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. 
Error Code: 0, 
SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000,
errorMessage:AnalysisException: Syntax error in line 1:
...table1 ("col_message_key" DECIMAL(12,0)...

I am expecting the code to create a table with the name 'table1'

Could you please help me with this?

noam621
  • 2,766
  • 1
  • 16
  • 26
MuraliKoti
  • 1
  • 1
  • you have to convert all the columns data type into impala compatible data type. – maogautam Aug 16 '19 at 02:26
  • When I use the df.printschema(), I see string as the data type. But the create table command as part of the SaveasTable() is converting that as TEXT data type. No idea why it is changing to TEXT.. any thoughts? – MuraliKoti Aug 19 '19 at 10:59
  • could you please check whether table you're creating is already exist in impala or not. If it is there then it won't drop table. It might be taking that schema definition. For simplicity just create a unique table and test it. – maogautam Aug 21 '19 at 19:53

0 Answers0