DataBricks Delta Live Tables Expectations: How to dynamically execute @dlt.expect()

Question

I tried the following code, but results from running DLT pipeline in an error:

if kwargs.get("df_tableoperation", None) is not None : ^ SyntaxError: invalid syntax

The idea is to dynamically generate a series of tables from configurable metadata tables. Where I would ultimately keep rule sets that are applied conditionally.

I cannot picture how to accomplish this, would I need to explicitly declare all of the @dlt.expect_<*> in the framework and pass in empty dicts if I did not want to apply any rules?

def outer_create_bz_table( **kwargs )

    @dlt.table(name = bz_tablename, comment="test")
    

    if kwargs.get("df_tableoperation", None) is not None :

        @dlt.expect_all_or_drop({"Null Location" :  "LOCATION is null"}) 

I was expecting this to run as a DLT pipeline.

Ultimately I want to dynamically call any number of @dlt.expect_<suffix> to operate on any number of columns for respective tables generated with the outer_create_bz_table(). I got some confirmation of a generalized framework being feasible when I tried the same dynamic calling of if kwargs.get() : dlt.apply_changes(someDynamicArg). The only differences between that and with the above approach are two:

1. Expectations make use of the @ decorator
2. Expectations occur before the function def that returns a df

(Edit) Looked more into the idea of decorators, I've also tried the following. But results were still a syntax error but pointing out x = a_function_to_decorate()

def expect_decor(a_function_to_decorate):

    def the_wrapper_around_the_original_function():

        @dlt.expect_all_or_drop({"Null Location" :  "LOCATION is null"}) 
        x = a_function_to_decorate()
        return x
    
    return the_wrapper_around_the_original_function

# And then 

def outer_create_bz_table( **kwargs )

    @dlt.table(name = bz_tablename, comment="test")
    @expect_decor
    def : return df # some df

I do not see any reference to accomplishing this in the documentation on multiple expectations:

https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/expectations#--multiple-expectations

Answer 1

The error is because you can't give any statements after dlt.table decorator. You need to give the def functions which returns dataframe. But to achieve your task conditionally you can use one of the below ways.

Create the dlt table inside if condition.

import pandas as pd
import dlt
    
def outer_create_bz_table( **kwargs ):
    if kwargs.get("df_tableoperation", None) is  not  None:
         @dlt.table(name = "pd_data", comment="test")
         @dlt.expect_all_or_drop({"Null Location" : "LOCATION is null"})
         def get_sample():
              dt = [
                  {
                      "LOCATION":"Bengaluru",
                      "id":"123"
                   },
                   {
                       "LOCATION":None,
                       "id":"456"
                    }
                ]
               return spark.createDataFrame(pd.DataFrame(dt))

and call the function.

outer_create_bz_table(df_tableoperation="yes")

After run pipeline below is the output.

Or

you create dictionary of rules based on condition and pass that dictionary object to except_all function.

import pandas as pd
import dlt

def outer_create_bz_table( **kwargs ):
    rules={}
    if kwargs.get("df_tableoperation", None) is  not  None:
        rules["Null Location"]="LOCATION is null"
        
    @dlt.table(name = "pd_data", comment="test")
    @dlt.expect_all_or_drop(rules)
    
    def get_sample():
        dt = [
                    {
                    "LOCATION":"Bengaluru",
                    "id":"123"
                    },
                    {
                    "LOCATION":None,
                    "id":"456"
                    }
                ]
        return spark.createDataFrame(pd.DataFrame(dt))

Code in notebook.

Output;