0

Below is the code that I am trying in PySpark

from delta import DeltaTable

delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.logRetentionDuration = "interval 1 days"

After this do we need to save this config or it will be applicable automatically. How we can check what is current logRetentionDuration set for table. I tried below to get properties info

delta_table.detail()

But it return empty {}

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
YOGESH
  • 43
  • 5

1 Answers1

1

Use Spark SQL:

spark.sql("ALTER TABLE delta.`path\to\delta\table` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")

spark.sql("DESCRIBE DETAIL delta.`path\to\delta\table\path`").show(truncate=False)

Under the "properties" column you can see the log retention duration of the specified delta table.

Note: If the retention period was not set properly, the above SQL command returns {} in the properties column.

arudsekaberne
  • 830
  • 4
  • 11
  • I did that it return {} in properties column. – YOGESH Mar 15 '23 at 10:31
  • Try this `spark.sql("ALTER TABLE delta.\`path\to\delta\table\` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")` and then use the decribe command. – arudsekaberne Mar 15 '23 at 10:36
  • Thanks it set property. I accept answer. Is there any API option to do the same using DeltaTable object. – YOGESH Mar 15 '23 at 11:15