Remove folder in blob in a more effective way

Question

Suppose I am going to remove a folder from the blob from databricks. however when the folder is not empty , it throws the error "non-empty" directory is not permitted using dbutils.fs.rm("wabs://..../folder") Even has error include recurse=True How to solve it?

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

0

Note: Delete the file or directory (optionally recursively delete all files in the directory). This call throws an exception with IO_ERROR if the path is a non-empty directory and recursive is set to false or on other similar errors.

For more details, refer similar SO thread.

Hope this helps.

edited Jun 20 '20 at 09:12

Community

1
1

answered Oct 30 '19 at 04:30

CHEEKATLAPRADEEP

12,191
1
19
42

so rm a directory recusively is not permitted on non-empty directory ? – mytabi Oct 30 '19 at 06:25
The rm function receives 2 parameters : rm(dir: String, recurse: boolean = false): boolean -> Removes a file or directory Where the second parameter is a boolean flag to set the recursitivity, so you just need to set it to true: – CHEEKATLAPRADEEP Oct 30 '19 at 06:31
finally I use shutil.rmtree() to remove the folder. and create back using dbutils.fs.mkdirs – mytabi Oct 31 '19 at 05:04
hi @mytabi , I encounter `No such file or directory` error when using `shutil.rmtree()` instead of `dbutils.fs.rm()` on `wasbs://`-prefixed path, Could you describe how you did it on `wasbs://`-prefixed path? Thanks! – johnnyasd12 Jun 16 '22 at 13:21

score 0 · Answer 2 · answered Jun 16 '22 at 15:23

According to here, the below function helps me solve this problem.

def delete_mounted_dir(dirname):
    files=dbutils.fs.ls(dirname)
    for f in files:
        if f.isDir():
            delete_mounted_dir(f.path)
        dbutils.fs.rm(f.path, recurse=True)

Remove folder in blob in a more effective way

2 Answers2