In databricks Scala, I'm exploding a Map column and loading it into a delta table. I have a predefined schema of the delta table.
Let's say the schema has 4 columnns A,B,C,D.
So,on day 1 Im loading my dataframe with 4 columns into the delta table using the below code.
loadfinaldf.write.format("delta").option("mergeSchema", "true")\
.mode("append").insertInto("table")
The columns in dataframe changes everyday. For instance on day 2,two new columns E,F are added and there is no C column. Now I have 5 columns A,B,D,E,F in the dataframe. When I load this data into delta table, columns E and F should be dynamically created in the table scehma and the corresponding data should load into these two columns and column C should be populated as NULL. I was asuming that spark.conf.set("spark.databricks.delta.schema.autoMerge","true") will do the job. But Im unable to acheive this. Could someone please help out.
