0

How to delete a Partition in unmanaged/external delta lake table?

val deltaTable = DeltaTable.forName("country_people")
val partitionColumn = "country"
val partitionValue = "Argentina"

// Delete the partition data
val deltaTable = DeltaTable.forPath("dbfs:/user/hive/warehouse/country_people")
deltaTable.delete(condition = expr(s"$partitionColumn = '$partitionValue'"))
val retentionHours = 0 // Set the retention period to 0 hours to remove all deleted files
deltaTable.vacuum(retentionHours)

Will it be enough or do we also need to use dbutils.fs.rm("<your-storage-path>", true)

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Raj Aryan Pathak
  • 11
  • 3

1 Answers1

0

It's just enough to perform delete operation on partition, then vacuum should remove all referenced files. It's better not to use dbutils.fs.rm as you may corrupt the table.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132