5

quick question about Python on windows. I have a script that compiles a program (using an install rule), and then moves the build products to a remote destination over the network.

However, I keep getting WindowsError 5 Access Denied. All files are created from the script context, I have ownership and full control on all of them. The copy to the remote destination succeeds, but the failure is during the deletion process. If I try to delete or rename the file manually within windows, I get no errors. Just the shutil.move fails.

I'm thinking maybe the API is trying to delete the file when the network operation is not yet complete?

Any input is very appreciated.

try:
    shutil.move(directory, destination)
except OSError:
    print "Failed to move %s to %s." %(directory, destination)
    raise

...

Traceback (most recent call last):
  File "C:\WIP\BuildMachine\build_machine.py", line 176, in <module>
    main()
  File "C:\WIP\BuildMachine.hg\BuilderInstance.py", line 496, in deployVersion
    shutil.move(directory, destination)
  File "C:\Python27\lib\shutil.py", line 300, in move
    rmtree(src)
  File "C:\Python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 5] Access is denied: '3_54_7__B_1\\Application_Release_Note.doc'
shayst
  • 267
  • 4
  • 14

1 Answers1

5

the problem with shutil.move on windows is that it doesn't handle the case where:

  • the source & destination aren't on the same drive AND
  • some files in the source directory are write-protected.

If both conditions are met, shutil.move cannot perform a os.rename, it has to:

  • copy the files (which isn't an issue)
  • delete the source files (which is an issue because of a limitation of shutil)

To fix that, I made myself a copy of the shutil module (under a different name) and added that line (for you it'll be right before line 250):

   os.chmod(fullname,0o777)  # <-- add that line
   os.remove(fullname)  # some versions have "unlink" instead

The rmtree function has the same issue on Windows.

On Linux this doesn't happen because file delete permissions aren't handled at the file level but at the directory level. On windows, it doesn't work that way. Adding os.chmod does the trick (even if it's a hack), and os.remove succeeds (unless the file is open in Word or whatever)

Note that shutil authors encourage you to copy & improve the functions. Also a note from the documentation of shutil.move:

A lot more could be done here... A look at a mv.c shows a lot of the issues this implementation glosses over.

If you don't want to modify shutil, you can run a recursive chmod on the source files to make sure that shutil.move will work, for instance like this:

for root, dirs, files in os.walk(path):  
  for f in dirs+files:  
    os.chmod(os.path.join(root, f), 0o777)

You could also use shutil.copytree then a modified version of shutil.rmtree (since you know that source & dest aren't on the same filesystem)

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • The readonly file attribute is not a permission and actually has no business being managed in `chmod` except for legacy weirdness in the Windows C runtime. It's approximately equivalent to the Linux [i]mmutable file attribute that root can set via `chattr +i filename` (note: *not* `chmod`), except the immutable flag is honored for the contents of a directory in Linux, while in Windows readonly only prevents the directory itself from being deleted, not its contents. Also, in Windows filesystems with security (NTFS) it's not uncommon to grant non-admin users the right to set file attributes. – Eryk Sun Jun 29 '18 at 03:04
  • Also, in Windows filesystems with security, delete access has nothing to do with permission to write to a file or a directory. There's a separate right for deleting. Also, the file's parent directory can grant the delete-child right to allow deleting the file even if the file security denies delete access to the user. All that said, if the file is currently open without delete sharing, trying to delete it will fail with a sharing violation regardless of permissions or attributes. – Eryk Sun Jun 29 '18 at 03:12
  • Thanks so much. It seems you are right, the file is in fact "read only" which is weird in itself. Your answer helped to solve the issue – shayst Jun 29 '18 at 04:45
  • @eryksun sorry if my answer was inaccurate. I think you should post an answer of your own. The main point is that `shutil` is designed with Linux in mind (with symlinks and all) but Windows is kind of left over. The `chmod` method works (I have tested it on my machine prior to posting), but I'm talking for experience. I didn't dig in windows file permissions It works, not sure that it's portable because on Linux you can delete someone else's file but not perform a chmod on it. – Jean-François Fabre Jun 29 '18 at 05:04
  • No reason to apologize. I was just explaining the situation and fleshing out the details of Windows delete permissions. Using `chmod` for this in Windows is an old hack that we have to live with. The Linux FAT driver even implements it, but not the NTFS driver, so it's not consistent. Linux has specific `fatattr` for FAT attributes and `setfattr` with "system.ntfs_attrib_be" for NTFS. Unfortunately there's no generic integration with `chattr`, unlike BSD and MacOS, in which `chflags` maps some DOS attributes such as immutable <=> readonly. – Eryk Sun Jun 29 '18 at 06:26
  • Also, in Linux this very same thing can happen, and even more so since whole directories can be made immutable, and `chmod` wouldn't help. The difference is that typically only root has the right (CAP_LINUX_IMMUTABLE) to mark a file as immutable in Linux. – Eryk Sun Jun 29 '18 at 06:28