I can easily set this option on databricks, but how to do the same with the local pyspark and delta.spark library
I've tried to set this option in sparkConf, but it doesn't work
("spark.databricks.delta.logRetentionDuration", "interval 52 weeks")
Asked
Active
Viewed 230 times
0
Col1ns
- 25
- 4
-
How are you confirming it doesn't work? This config is respected and should work. – Nick Karpov Nov 29 '22 at 17:02
-
I've resolved this problem, and the solution was to use spark.databricks.delta.properties.defaults.logRetentionDuration instead spark.databricks.delta.logRetentionDuration – Col1ns Nov 30 '22 at 13:17