shutil copy command writes 0 byte file copying temp files

Question

I have another method calling this function 3 times to write 3 different files. The first 2 files are copied to the destination folder as expected, the third file is always zero bytes. If I turn off deletion I can see that all 3 temp files are written to successfully. The code reports no errors, but at the end of the process the third file is always empty.

Does anyone know why this is failing 1 out of 3 times?

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile() as fp:
        fp.write(content)
        shutil.copy(fp.name, file_destination)

The following variation works, however, I would like to understand why the first two files work and the third one does not in the above code.

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile(delete=False) as fp:
        fp.write(content)
    shutil.copy(fp.name, file_destination)
    os.remove(fp.name)

tyrion · Accepted Answer · 2019-03-19T08:59:50.837

This is because the content has not been written to disk yet, when you perform the copy. This happens because writes are buffered, and do not always happen immediately after you call file.write. To make sure the content is written to disk at a given point, you can use file.flush.

In your case, it is sufficient to change your code to:

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile() as fp:
        fp.write(content)
        fp.flush()
        shutil.copy(fp.name, file_destination)

For more information on when the contents are actually written to disk you can see the documentation of io.BufferedWriter. The relevant part is:

The buffer will be written out to the underlying RawIOBase object under various conditions, including:

when the buffer gets too small for all pending data;

when flush() is called;

when a seek() is requested (for BufferedRandom objects);

when the BufferedWriter object is closed or destroyed.

Therefore, in your first example, it is possible that it works only sometimes, because those times the content you are writing exceeds the buffer and so needs to be written out immediately.

Your second example, instead, works, because when you are exiting the with block, the file is closed and thus the buffer needs to be flushed out and written to disk.

Note that [the docs](https://docs.python.org/3/library/os.html#os.fsync) recommend to use `os.fsync(fd)` after invoking `flush()` of the file object. `flush` clears Python's file object buffer, `fsync` instructs the OS to write its file system buffers corresponding to the filedescriptor to disk. — shmee, Mar 19 '19 at 09:03
Thanks, this makes sense, the first two files were larger, so the third being smaller probably meant it did not cross a flush boundary... — Andrew Gill, Mar 19 '19 at 09:16

shutil copy command writes 0 byte file copying temp files

1 Answers1