0

I'm trying to execute VACUUM in Synapse notebook, and I'm using this code:

spark.sql("SET spark.databricks.delta.retentionDurationCheck.enabled = false")

#get list of all tables
    table_list = spark.sql("show tables from db").select('tableName').rdd.map(lambda x : x[0]).collect()
     
    for table in table_list:
      spark.sql("VACUUM db.`{0}`".format(table)).show()

I think it's not running properly because I still have more than 1 billion rows in raw layer for example.

The code runs successfully, but if I execute a DESCRIBE HISTORY I verify that the operation does not appear in the history.

Does anyone have similar issue/experience or have an idea if I'm configuring it correctly or if it's not supported by synapse workspace?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
coding
  • 135
  • 2
  • 9
  • Do you get any error, or why do you think that it doesn't work? – Alex Ott Aug 25 '23 at 10:37
  • @AlexOtt the code runs successfully, but if I execute a DESCRIBE HISTORY I verify that the operation does not appear in the history. – coding Aug 25 '23 at 17:24
  • most probably you don't have data old enough (default retantion is 7 days) – Alex Ott Aug 25 '23 at 18:52
  • I think is not that problem, I have at least 3 years of data. But thanks for your help. – coding Aug 25 '23 at 19:09
  • 3 years of data doesn’t mean you have something to delete if it was append only without optimize/delete/update/merge operations – Alex Ott Aug 25 '23 at 20:43

0 Answers0