Explorer has an open handle to the directory that shares delete/rename access. This allows rmdir to succeed, whereas normally an open would not share delete/rename access, and rmdir would fail with a sharing violation (32). However, even though rmdir succeeds, the directory doesn't actually get unlinked until Explorer closes its handle. It's watching the directory for changes, so it gets notified that the directory has been deleted, but even if it closes its handle immediately, there's a race condition with the script's os.mkdir call.
You should retry os.mkdir in a loop, with an increasing timeout. You also need an onerror handler for shutil.rmtree that handles trying to remove a directory that's not empty because it contains 'deleted' files or directories.
For example:
import os
import time
import errno
import shutil
def onerror(function, path, exc_info):
# Handle ENOTEMPTY for rmdir
if (function is os.rmdir
and issubclass(exc_info[0], OSError)
and exc_info[1].errno == errno.ENOTEMPTY):
timeout = 0.001
while timeout < 2:
if not os.listdir(path):
return os.rmdir(path)
time.sleep(timeout)
timeout *= 2
raise
def clean_dir_safe(path):
shutil.rmtree(path, onerror=onerror)
# rmtree didn't fail, but path may still be linked if there is or was
# a handle that shares delete access. Assume the owner of the handle
# is watching for changes and will close it ASAP. So retry creating
# the directory by using a loop with an increasing timeout.
timeout = 0.001
while True:
try:
return os.mkdir(path)
except PermissionError as e:
# Getting access denied (5) when trying to create a file or
# directory means either the caller lacks access to the
# parent directory or that a file or directory with that
# name exists but is in the deleted state. Handle both cases
# the same way. Otherwise, re-raise the exception for other
# permission errors, such as a sharing violation (32).
if e.winerror != 5 or timeout >= 2:
raise
time.sleep(timeout)
timeout *= 2
Discussion
In common cases, this problem is 'avoided' because existing opens do not share delete/rename access. In this case, trying to delete a file or directory fails with a sharing violation (winerror 32). For example, if a directory is open as the working directory of a process, it doesn't share delete/rename access. For regular files, most programs only share read/execute and write/append access.
Temporary files are often opened with delete/rename access sharing, especially if they're opened with delete/rename access (e.g. opened with the delete-on-close flag). This is the most common cause of 'deleted' files that are still linked but are inaccessible. Another case is opening a directory to watch for changes (e.g. see ReadDirectoryChangesW). Typically this open will share delete/rename access, which is the situation with Explorer in this question.
Stating that a file gets deleted without getting unlinked probably sounds strange (to say the least) to a Unix developer. In Windows, deleting a file (or directory) is just setting a delete disposition on its file control block (FCB). A file that has its delete disposition set gets automatically unlinked when the filesystem cleans up the file's last kernel file-object reference. A file object typically gets created by CreateFileW, which returns a handle to the object. Cleanup of a file object is triggered when the last handle to it is closed. Multiple handle references for a file object may exist due to handle inheritance in child processes or explicit DuplicateHandle calls.
To reiterate, a file or directory may be referenced by multiple kernel file objects, for which each may be referenced by multiple handles. Normally, with classic Windows delete semantics, all of the handles have to be closed before the file gets unlinked. Moreover, setting the delete disposition isn't necessarily final. If any of the open handles has delete/rename access, it can actually be used to restore access to the file by clearing the delete disposition (e.g. see SetFileInformationByHandle: FileDispositionInfo).
In Windows 10, the kernel also supports POSIX delete semantics, with which a file or directory is immediately unlinked as soon as the deleting handle is closed (see the details for NTAPI FileDispositionInformationEx). NTFS has been updated to support POSIX delete semantics. Recently WINAPI DeleteFileW (i.e. Python os.remove) has switched to using it if the filesystem supports it, but RemoveDirectoryW (i.e. Python os.rmdir) is still limited to a classic Windows delete.
Implementing POSIX semantics is relatively easy for NTFS. It simply sets the delete disposition and renames the file out of the way into an NTFS reserved directory "\$Extend\$Deleted", with a name that's based on its file ID. In practice, it appears that the file was unlinked, while continuing to allow existing file objects to access the file. One significant difference compared to a classic delete is that the original name is lost, so the delete disposition cannot be unset by existing handles that have delete/rename access.
