0

I have found a ton of examples showing how to Merge data using Databricks Delta Table Merge to load data to SQL DB. However, I'm trying to find examples whereby trying to load data to SQL DB without Databricks Delta Merge fails.

This is because I'm having trouble getting my head around knowing a situation where I should be using Databricks Delta Merge.

Therefore, can someone point me to a link showing where loading data to SQL DB from Databricks would fail withou Databricks Delta Merge, alternatively steps I would have to take to merge without Databricks Delta Lake Merge?

Carltonp
  • 1,166
  • 5
  • 19
  • 39
  • It's not that merge is absolutely necessary but it can make your notebook more efficient. If you have a lot of data, it takes a lot of time to create that table, it's like indexing fields or something. I'm in a situation where I need to ingest new data in parquet files and it would take too long to create the delta table for all the data with each new request so I'm going to need this merge command or some other data ingestion technique. – Dudeman3000 Mar 04 '21 at 19:22
  • Can we use delta merge [destDataFrame.merge(sourceDataFrame)] along with `Df.write.format("delta").option("mergeSchema", "true").mode("overwrite").partitionBy("id").save(DestFolderPath)` – Antony Aug 19 '21 at 12:01

0 Answers0