1

I have a tables that travels from Bronze - silver - gold,

I want to implement some function like 'is_full_refresh()' so the pipeline filters the df depending on the output, if it's a full, don't filter, if it's incremental filter by a,b,c

Checking the documentation on Databricks https://docs.databricks.com/delta-live-tables/settings.html#cluster-config can't find a direct way to differentiate between a full refresh and an incremental,

How can I do that?

Leonardo Lima
  • 83
  • 1
  • 7

1 Answers1

1

One option would be to rely on REST API call https://docs.databricks.com/api/workspace/pipelines/getupdate.

Another option I would try (requires some investigation) is to query DLT event logs: https://docs.databricks.com/delta-live-tables/observability.html. I guess some log events may have this info.

partlov
  • 13,789
  • 6
  • 63
  • 82
  • To add to the answer, /api/2.0/pipelines/{pipeline_id}/updates have the response "full_refresh": true which is exactly what I was looking for – Leonardo Lima Jul 24 '23 at 12:15