In our DLT gold layer we have some aggregation queries that are live so it computes the whole thing. We would like to make this quicker and use CDF for business level aggregates like below
https://www.databricks.com/notebooks/delta-lake-cdf.html
We are thinking along the lines of only computing aggregations for the changes and merging to gold layer . for which I need to find out what has changed in a delta table after a dlt pipeline has run and then do aggregations for the changes and merge to the gold layer?Similar to the link (https://www.databricks.com/notebooks/delta-lake-cdf.html)
From what I see DLT overwrites the whole table? How can I accomplish this using DLT
Kind Regards Priya
