How do I combine two files without overwriting any data in python?

Question

I need to concatenate two files, one which contains a single number and the other which contains at least two rows of data. I have tried shutil.copyfile(file2,file1) and subprocess.call("cat " + file2 + " >> " + file1, shell=True), both things give me the same result. The file with the single number contains an integer and a newline (i.e. two characters) so when I bring the two files together the first two characters of file2 are overwritten instead of just added to the end. If I do it through the shell using "cat file2 >> file1" this does not happen and it works perfectly.

Here is what I mean:

import numpy as np
from subprocess import call

f.open(file1)
f.write('2\n')
np.savetxt(file2,datafile,fmt)
call("cat " + file2 " >> " + file1, shell=True)

So instead of getting:

2
data data data ...
data data data ...

I get:

2
ta data data ...
data data data ...

I have no idea what is causing this issue but it is very frustrating. Any suggestions?

Have you tried `file1` in `a` mode and simply writing `file2` to `file1`? — Maciej Gol, Oct 18 '13 at 22:20

score 1 · Answer 1 · answered Oct 18 '13 at 22:20

1

Have you tried closing file1 first?

 f.close()
 np.savetxt... Etc

answered Oct 18 '13 at 22:20

beroe

11,784
5
34
79

score 1 · Accepted Answer · answered Oct 18 '13 at 23:49

1

The problem is that you haven't flushed f. "2\n" is still in the file buffer and overwrites the other data after cat completes when f is eventually closed. But there Is a better way to do this. Reading the numpy docs savetxt, you can pass in a file handle. Numpy can use the existing file handle to write its data. No need for a second temporary file.

import numpy as np

with open(file1, "w") as f:
    f.write('2\n')
    np.savetxt(f, datafile, fmt)

answered Oct 18 '13 at 23:49

tdelaney

73,364
6
83
116

THIS. This is EXACTLY what I wanted! I totally gave up on trying to write the integer then writing the data and figured it would be easier to concatenate the two files! Thank you! I would upvote if I could but my reputation isn't high enough. Thank you though! – spacemonkey5665 Oct 19 '13 at 06:31

score 0 · Answer 3 · edited May 23 '17 at 12:21

To append file2 to file1, you could use 'a' file mode as @krookik suggested:

from shutil import copyfileobj

with open(file1, 'w') as f: # write file1
    f.write('2\n')
# `with`-statement closes the file automatically

# append file2 to file1
with open(file2, 'rb') as input_file, open(file1, 'ab') as output_file:
    copyfileobj(input_file, output_file)

Your code doesn't work because it might be missing f.flush() or f.close() after f.write('2\n') as @beroe suggested i.e., when the cat command appends to file1, its content is not flushed from memory to disk yet, '2\n' is written later (when the file is closed implicitly on the program exit) and therefore it overwrites content written by cat.

How do I combine two files without overwriting any data in python?

3 Answers3