1

I am trying to write simple spark dataframe to db2 database using pySpark. Dataframe has only one column with double as a data type.

This is the dataframe with only one row and one column: PySpark dataframe

This is the dataframe schema: PySpark dataframe schema

When I try to write this dataframe to db2 table with this syntax:

dataframe.write.mode('overwrite').jdbc(url=url, table=source, properties=prop)

it creates the table in the database for the first time without any issue, but if I run the code second time, it throws an exception: Error Log

On the DB2 side the column datatype is also DOUBLE.

Not sure what am I missing.

laughedelic
  • 6,230
  • 1
  • 32
  • 41
vaibhav sapkal
  • 11
  • 2
  • @pault : but actually with mode ='overwrite' expected behavior is , if data/table already exists, existing data is expected to be overwritten by the contents of the DataFrame. In this case the PySpark dataframe schema and db2 table schema are matching so ideally overwrite should not give issue. Also mode = 'append' is not working in this case. – vaibhav sapkal Mar 20 '18 at 19:24

1 Answers1

0

I just changed small part in the code and it worked perfectly. Here is the small change I did to syntax

dataframe.write.jdbc(url=url, table=source, mode = 'overwrite', properties=prop)
vaibhav sapkal
  • 11
  • 2