DLT - merge/apply_changes using partition pruning

Asked Jun 08 '23 at 18:53

Active Jun 08 '23 at 19:04

Viewed 140 times

Is there any way we can scan the target table partitions and apply merge in only those partitions using delta live table?

I have a dlt (delta live table) table created with partitions (reference : https://docs.databricks.com/delta-live-tables/python-ref.html#control-how-tables-are-materialized) , however I did not find anything which confirms that the partitions are pruned when using dlt.apply_changes.

Ideally I would want to scan the source for a list partition values from my source query and bring in only those partitions from the target delta table to be scanned instead of the whole delta table when applying the merge.

Any suggestions if this can be done using delta live tables?

edited Jun 08 '23 at 19:04

asked Jun 08 '23 at 18:53

anky

74,114
11
41
70

Are you expecting the target to be as partioned or just the part files? – DileeprajnarayanThumula Jun 09 '23 at 07:06
@DileeprajnarayanThumula target will be partitioned on a column. I am trying to scan only those partitions and underlying files that are present in my source query and disregard the rest of the partitions when merging – anky Jun 09 '23 at 07:25
Could you please help me with source query. – DileeprajnarayanThumula Jun 09 '23 at 07:59

DLT - merge/apply_changes using partition pruning

0 Answers0