2

I have a parameter value which determines whether the table needs to be full load or an incremental load. In delta live tables, incremental load is not an issue as we apply changes and specify whether the table needs to be SCD1 or SCD2.

However, there are scenarios where the table is not an incremental table. Every time the data comes in, the silver table needs to be truncated and reloaded again completely. I tried setting the value apply_as_truncates =True, but this doesn't work.

dlt.apply_changes(
  source=bronze_tablename,
  target=silver_tablename,
  keys=key_fields,
  sequence_by=sequence_by, 
  apply_as_truncates = True,
  stored_as_scd_type=1
 )`

Could you please let us know how a DLT in silver can be truncated and reloaded?

Thanks in advance.

Regards, R

RLH
  • 35
  • 4

1 Answers1

1

You need to declare this table as LIVE table, not STREAMING LIVE table - in this case it will be recalculated each time new data is coming

Alex Ott
  • 80,552
  • 8
  • 87
  • 132