1

I have a delta table with 5 partitions, one of the partition being a runid column. When I try to delete using the runid underlying parquet files gets deleted, after using vacuum command. But this does not remove the runid partition. If I run the same vacuum command 4 times then it removes the runid partition.

For Hive we have drop partition but the same was not applicable for delta table!

Is this how the delete works in Delta table or is there any better way to delete the data and the partition for a runid from a managed delta table?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132

1 Answers1

0

Apart from runid, do you have two more level of partitioning. From what I have tried and tested so far, I understand after running first vacuum (which deletes the files), each consequent vacuum run only drop one level of directory. For ex if partition structure is such as: FolderA > FolderB > FolderC Then one first vacuum call will delete only the files inside FolderC, second call will drop empty FolderC then next call will drop FolderB and so on.

tikiabbas
  • 119
  • 2
  • 3
  • 11