0

I've created a Python script to help manage my media on an unRAID server. New files are automatically transferred into a folder and the script uses the folder structure and filenames to determine where they should be moved, usually into another share on the same drive. With Midnight Commander, this transfer happens instantly. This process in Explorer or my script using shutil.move() copies the file then deletes the original, which can take a long time for big files and also creates unnecessary read/writes on the drive. It's instant in both Explorer and Python when the transfer is on the same share.

Simplified pseudocode - Transfer and TV are shares on the same disk in this example:

from shutil import move

oldPath = r'\\NAS\Transfer\incoming\test.mkv'
newPath = r'\\NAS\TV\test.mkv'

move(oldPath, newPath)

Is there a way to move the file's pointer with Python, as I assume MC is doing, instead of physically moving the file?

TylerH
  • 20,799
  • 66
  • 75
  • 101
tekowalski
  • 19
  • 4
  • `os.rename()` is closer to the underlying OS layer than `shutil` is. On a standard UNIXy system, `rename()` is atomic if-and-only-if both source and destination are on the same filesystem. – Charles Duffy Mar 07 '21 at 03:59
  • ...I'm _guessing_ that your source and destination are truly on the same filesystem, but they don't report that correctly in a way that causes `_samefile()` to return true. See the source code to `shutil.move` in https://github.com/python/cpython/blob/2c0a0b04a42dc4965fcfaef936f497e44f06dea5/Lib/shutil.py#L764-L828 – Charles Duffy Mar 07 '21 at 04:02
  • Thanks @CharlesDuffy. I originally used os.rename() but, although the shares are actually on the same drive, it fails with "The system cannot move the file to a different disk drive." – tekowalski Mar 07 '21 at 04:02
  • The `rename()` syscall is the one used to ask the OS do a move (to a new name within the same filesystem) without a copy. If your OS is refusing to do it, that's a problem with your OS. (It's curious that you have other software that succeeds here. Is your copy of MC a native Windows app, or is it working through a translation layer like Cygwin or SFU? And similarly: is the Python interpreter that's failing native? I'm wondering if there's a translation layer in play that's either fixing a compatibility bug, or making an assumption that fails to hold with the network filesystem in play) – Charles Duffy Mar 07 '21 at 04:03
  • @CharlesDuffy Midnight Commander is a file manager that runs on the Linux server, typically through SSH for me. It's very robust and efficient, especially for my use case. If there isn't a native solution available, I'll probably try rewriting the shutil library to incorporate a manual check to bypass the limitation. Appreciate the help! – tekowalski Mar 07 '21 at 04:08
  • 1
    Are you running your _Python_ program "on the Linux server", or on a client? If the Python program works on Linux, you know it's a problem with the network filesystem layer -- not a difference between the applications, but a difference relating to _where you run them_ in each of the two cases. – Charles Duffy Mar 07 '21 at 05:25
  • `shutil` is just calling `os.rename()` as/when possible; if that fails, you won't be able to override it successfully. – Charles Duffy Mar 07 '21 at 05:26
  • You were exactly right - the issue was that the Python program was running on Windows, which was unable to tell that the disks were the same. When I moved the script to the server and changed the path structure, it worked as expected immediately. Thank you! If you post an answer with any similar info, I'd be happy to accept it as the answer. – tekowalski Mar 07 '21 at 05:36

1 Answers1

1

The underlying interface for giving a file a new name while keeping it on the same filesystem is os.rename(). If that call fails, an in-place rename will not be possible.

If you tested that an efficient local rename is possible directly on the file server, but not over your network share, you may be disregarding a limitation of your network filesystem (or how it's configured; make sure both source and destination are under the same mount point from the client's perspective, not just the server's).

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • The issue was that the script was running from Windows, not the server. Operations initiated from the server occur instantly. However, in testing I found that these operations ignore the "split level" setting of the share and always remain on the same drive. I'll be writing a drive test to determine whether it should rename (same drive) or move (allowing change of drive). Thanks for your help – tekowalski Mar 07 '21 at 14:53