0

New to databricks and spark, I'm trying to run the below command and met this error

spark.databricks.delta.retentionDurationCheck.enabled= "false"

error: 'SparkSession' object has no attribute 'databricks'

efsee
  • 579
  • 1
  • 10
  • 22
  • 1
    This looks like a configuration. If its a SQL configuration, use `SET` command , otherwise use `--conf` to pass it to the Spark application – DaRkMaN Aug 31 '19 at 11:56

1 Answers1

5

It should be as sql

spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled=false")

Or put it in as spark Conf property

SMaZ
  • 2,515
  • 1
  • 12
  • 26
  • Could you provide an example of specifying such a spark conf property? – Samuel Lampa May 04 '23 at 11:06
  • @SamuelLampa - See if this helps : https://community.databricks.com/s/question/0D53f00001mIUHACA4/how-to-set-retention-period-for-a-delta-table-lower-than-the-default-period-is-it-even-possible – SMaZ May 04 '23 at 15:57
  • 1
    Many thanks @SMaZ . Yes, the syntax that in the above case would be: `spark.conf.set("databricks.delta.retentionDurationCheck.enabled", "false")` did indeed work! – Samuel Lampa May 10 '23 at 09:48