2
%python
dataframe.count()  
#output 1179  

%python
dataframe.write.mode("overwrite").saveAsTable("tablename")

%sql
select count(*) from tablename  
--output 1069

What can I be doing wrong? (these are different cells in databricks)

I want to overwrite the data. Dataframe has more rows, but is dropping some rows while writing into the table.

proutray
  • 1,943
  • 3
  • 30
  • 48

1 Answers1

0

in this above code, the existing data in the table will be overwritten with the data of the dataframe.if you want to keep the table data with the dataframe data in the table then you have to append the dataframe into the table.

dataframe.write.mode("append").saveAsTable("tablename")

This code will append your dataframe into the table.

Thank You!

jatin jain
  • 31
  • 2