0

I am building a Delta Live Tables pipeline and I need to perform complex quality checks on top of the data mart. Following the guidelines of DLT, this should be done in a separate temporary live table where the quality check logic is implemented as well as the expectations. I add the clause ON VIOLATION FAIL UPDATE on the temporary table with the quality check logic and I expect that when an expectation fails all the pipeline fails and all transactions within the execution get rollbacked. However, the behavior I see is that when the expectation fails only the final table fails but the transaction in the data mart table do not get rollbacked, as well as the other tables in the pipeline. In general, when an expectation with the clause ON VIOLATION FAIL UPDATE fails, only that table and the downstream tables fails and no transactions get committed but the other tables in the pipeline continue processing and commit transactions. Is there a way to set the behavior of DLT to immediately stop the execution and rollback any transaction ?

From https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/expectations Fail on invalid records When invalid records are unacceptable, use the expect or fail operator to stop execution > immediately when a record fails validation. If the operation is a table update, the system atomically rolls back the transaction

Houssem Guidara
  • 1

1 Answers1

0

It's expected behavior of Delta Live Tables as there is no support for multi-table transactions exist yet, so rollback happens only on the table where expectation has failed and pipeline will stop eventually.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Thank you for your answer, could you please recommend a way to implement the behavior I am looking for ? I tried to put the logic directly in the data mart table's expectations but it threw an error since apparently expectation expression does not support windowing or grouping. – Houssem Guidara Jul 18 '23 at 13:07
  • I would suggest to contact solution architect responsible for your Databricks account. There is no simple answer, but it all depends on details, etc. – Alex Ott Jul 18 '23 at 13:56