1

I want to copy the content of a file 'from_path' to the end of another file 'to_path'. I wrote the code

fd_from = os.open(from_path, os.O_RDONLY)
fd_to = os.open(to_path, os.O_WRONLY | os.O_APPEND)
os.copy_file_range(fd_from, fd_to, os.path.getsize(from_path))
os.close(fd_from)
os.close(fd_to)

However, I get the following error

OSError: [Errno 9] Bad file descriptor

on the third line.

This (or something similar) was working fine, but now I can't avoid said error, even though (I believe) I haven't changed anything.

I looked around online and figured that this error usually happens because a file was not properly opened/close. However, that should not be the case here. If we do, for example

fd_to = os.open(to_path, os.O_WRONLY | os.O_APPEND)
os.write(fd_to, b'something')
os.close(fd_to)

Everything works smoothly. Also, if I write the exact same code as the problematic one, but without O_APPEND, everything works as well.

I am using Python 3.8.13, glibc 2.35 and linux kernel 5.15.0.

Note that efficiency is important in my case, thus many of the alternatives I've came across are undesirable. Some of the alternatives that were found to be slower than this particular method are:

  • Using subprocess to launch the unix utility cat;
  • Iterating over the lines of the first file and appending them to the second.

While I had the implementation with copy_file_range working, I managed to find that this was around 2.6 times faster than cat and 14 times faster than iterating over the lines.

I've also read about shutil and other methods, but those don't seem to allow appending of the copied contents.

Can anyone explain the problem? Does this function not work with append mode? Or maybe there is a workaround?

Thank you in advance for your help!

Calhau18
  • 11
  • 1
  • 2
    The Linux manpage for `copy_file_range()` explicitly lists "The O_APPEND flag is set for the open file description(see open(2)) referred to by the file descriptor fd_out" as a cause for an EBADF error. I can find no explanation for why this limitation exists. You should be able to fix this by using the `offset_dst` parameter to start writing at the end of the file. – jasonharper Aug 18 '22 at 16:41
  • @jasonharper This is an unfortunate limitation, since there's no way to combine getting the destination file length and copying to the file atomically. But nothing in the copy_file_range description says it's atomic, so I guess this is just part of that. – Barmar Aug 18 '22 at 16:45

0 Answers0