shutil samefile error on Linux

Question

I have a python script that is giving me a hard time on Ubuntu 12.02 with Python 2.7.3.

PS: it runs without problems on Windows.

>>> import os
>>> import shutil

>>> shutil.copy("/mnt/my_network_dive/somewhere/sample.xml", "/mnt/my_network_drive/COMPLETED/")
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.7/shutil.py", line 117, in copy
        copyfile(src, dst)
    File "/usr/lib/python2.7/shutil.py", line 69, in copyfile
        raise Error("`%s` and `%s` are the same file" % (src, dst))
shutil.Error:    `/mnt/my_network_dive/somewhere/sample.xml` and `/mnt/my_network_drive/COMPLETED/sample.xml` are the same file

Checking some properties of the files:

>>> os.path.exists("/mnt/my_network_drive/somewhere/sample.xml")
True
>>> os.path.exists("/mnt/my_network_drive/COMPLETED/sample.xml")
True
>>> os.stat("/mnt/my_network_drive/somewhere/sample.xml")
posix.stat_result(st_mode=33272, st_ino=4913809333, st_dev=25L, st_nlink=1, st_uid=1000, st_gid=0, st_size=5447, st_atime=1465311674, st_mtime=1465311674, st_ctime=1465311685)
>>> os.stat("/mnt/my_network_drive/COMPLETED/sample.xml")
posix.stat_result(st_mode=33272, st_ino=4913809333, st_dev=25L, st_nlink=1, st_uid=1000, st_gid=0, st_size=10, st_atime=1465317482, st_mtime=1465317482, st_ctime=1465317483)
>>> os.path.islink("/mnt/my_network_drive/somewhere/sample.xml")
False
>>> os.path.islink("/mnt/my_network_drive/COMPLETED/sample.xml")
False

>>> shutil._samefile("/mnt/my_network_dive/somewhere/sample.xml", "/mnt/my_network_drive/COMPLETED/sample.xml")
False

As you see, calling shutil._samefile I get False but shutil.copy still raise the samefile error.

Am I forgetting something? Any other way to move or copy files with Python?

There's no need to copy them. They're two different directory entries pointing at the same file. Being the same file, there's no possible way their contents can differ. — Charles Duffy, Jun 07 '16 at 17:22
(Why `shutil._samefile` returns False is a different question, and perhaps something to file a bug over, but since `st_dev` and `st_ino` are identical, there's no question that it genuinely is the same content pointed to by both directory entries). — Charles Duffy, Jun 07 '16 at 17:24
@vmenezes The first argument of copy in the first box contains dive instread of drive, is it only typo? — Adam Przedniczek, Jun 07 '16 at 17:25
On Macintosh & Unix `shutil._samefile()` uses `os.path.samefile(src, dst)` which is only available on those platforms to determine if two paths are the same. On other platforms like Windows, it just compares the two normalized absolute path strings — which is likely part of why the results differ. — martineau, Jun 07 '16 at 17:33

Brendan Abel · Answer 1 · 2016-06-07T17:31:02.633

2

It looks like the two files are both hard links to the same file. You can tell because they share the same inode number

st_ino=4913809333

Windows users generally don't create hard links. They are more common in linux environments, which is why you may have not encountered the problem until now.

It is odd that samefile returns False. What OS are you using? shutil._samefile is just a wrapper around os.path.samefile (on systems where that function exists). What results do you get from os.path.samefile? On posix systems, it just checks that the device and inode match (which they do in your case) and it should return True

posixpath.py

def samefile(f1, f2):
    s1 = os.stat(f1)
    s2 = os.stat(f2)
    return samestat(s1, s2)

def samestat(s1, s2):
    return s1.st_ino == s2.st_ino and s1.st_dev == s2.st_dev

edited Jun 07 '16 at 17:31

answered Jun 07 '16 at 17:21

Brendan Abel

35,343
14
88
118

It's likely that one or both of the parent directories are symlinks to the same directory. – tdelaney Jun 07 '16 at 17:33
@tdelaney But `samefile` should still return `True` in that case, right? – Brendan Abel Jun 07 '16 at 17:34
Yes, I am as puzzled by that as you. As with a hard link, the dev and inode should be the same. – tdelaney Jun 07 '16 at 17:38
Thanks for the exnplanation @BrendanAbel I'm using Ubuntu 12.02 I ended up doing a `os.system("cp --remove-destination file1 file2")` to workaround it. – vmenezes Jun 07 '16 at 18:00
getting worse, there is another app that probably doesn't use context management well and sometimes it creates a new file on my `dest_folder` but dont save it and quit. When I try to copy from `src_folder` to `dest_folder` on another script with `shutil` I get the same errror, but if I go on shell `ls -la dest_folder` the folder is empty. Sounds crazy and probably I could blame the other script but `os.system("cp --remove-destination src_file destfile")` worked. – vmenezes Jun 07 '16 at 18:50

score 0 · Answer 2 · answered Jun 07 '16 at 17:33

0

In python 3 in shutil.copy there's additional argument follow_symlinks=True Look at this https://docs.python.org/3/library/shutil.html

If follow_symlinks is false, and src is a symbolic link, dst will be created as a symbolic link. If follow_symlinks is true and src is a symbolic link, dst will be a copy of the file src refers to.

answered Jun 07 '16 at 17:33

Adam Przedniczek

202
4
14

Doesn't answer OP's question about Python 2.7.3. – martineau Jun 07 '16 at 17:35
@martineau But maybe checking how it works in Python 3 could help determine if that's the problem. If yes, maybe he could look at the source of the newer version and copy this code. – Adam Przedniczek Jun 07 '16 at 17:39
How do you know it's not a problem with using Python 3 as well? – martineau Jun 07 '16 at 17:51
@martineau Dear Martin, I don't insist that it would solve the whole problem, but probably it has do with links. I only recommend to check both options for follow_symlink in Python 3. If with one of them it works properly we can always import this solution to Python 2.7 - we can always copy code and redefine this function. I suggested this transition to Python 3 and its version of shutil solely to diagnose the problem and presumable solution. Quote "How do you know it's not a problem with using Python 3 as well?" - I don't know because we don't have access to his network drive. – Adam Przedniczek Jun 07 '16 at 18:02
2

the problem was bigger then that, there was actually times that even when the dest_file didn't exist I was getting the same error(because another script using the file before crashed without closing the file, I could get out of python and run `ls -la` in the folder and there was no file in there but when copying it would break with `same file` error).My workaround was `os.system("cp --remove-destination src_file dest_file")`. – vmenezes Jun 07 '16 at 18:42
1

vmenezes: Glad you found a work-around...but it sounds like you still don't know what causes the problem when you try to use Python. – martineau Jun 07 '16 at 21:31

shutil samefile error on Linux

2 Answers2