1

I have the following code:

@dlt.table(
    name="ingested_data",
    comment="Ingest the table",
    table_properties={
        "quality": "raw", "name": "property_name"
    }
)

I am confused what the table_properties dictionary does in practice? I have seen no effect of the key-value pairs. Here is the list of the accepted keys.

Oliver Angelil
  • 1,099
  • 15
  • 31

1 Answers1

1

Table properties are used for two things:

  1. Defining specific configurations that will be used by the underlying engine - they are listed in documentation. Like, pipelines.reset.allowed defines if it's allowed to perform a reset on the given table (it's good idea to disable it on the table that holds data ingested from Kafka), or pipelines.autoOptimize.zOrderCols defines a list of columns that will be used in OPTIMIZE ZORDER BY <cols>.
  2. Allow to specify arbitrary table properties/metadata, like, data quality level (raw in your example`), tags, etc. that could be used when searching for tables in UI.

1st kind of usage happens often in practice, while 2nd is optional.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132