I have relatively big table and it is overwritten in DLT pipeline. It is partitioned by date and in most cases I change small portion of data (connected to last couple of partitions). Is it possible to selectively overwrite only specified partitions?
Asked
Active
Viewed 65 times
0
-
Is this what you mean? https://docs.databricks.com/delta/selective-overwrite.html – ferdyh Jan 10 '23 at 21:17
-
Yes it is, but I need confirmation if this works in DLT pipeline. I don't call `write` there and not sure how I would pass this option. – partlov Jan 11 '23 at 10:04