Quick write to a file for a Python subprocess

Question

I have a shell program that needs to be pointed at a text file. I would like to write to a temporary file, run this program on it via Python's subprocess method, and save the output. Here's what I have so far:

with open('test.txt', 'w') as f:
    for i in range(50):
        f.write('sometext %d'%i)
        f.flush()
        output = subprocess.check_output('./program test.txt', shell=True, stderr=subprocess.STDOUT)

Output only gets updated once. I'm assuming that flush isn't able to write to the file before the next iteration of the loop executes. What other ways could this be implemented? Opening and closing the file seems like it would be expensive, because this code is going to be called many times.

score 1 · Accepted Answer · answered Jun 18 '14 at 12:58

1

If you want to omit closing and re-opening (which would probably be the cleanest way), you might want to f.seek(0) at the start and f.truncate() after writing.

Otherwise, the data is appended to the file on every loop run and the external program might be confused.

answered Jun 18 '14 at 12:58

glglgl

89,107
13
149
217

This seems to work the best among the given options. – tlnagy Jun 18 '14 at 13:21

score 0 · Answer 2 · edited May 23 '17 at 11:49

0

f.flush will not definitely write to disk, if buffer isn't filled.

Check this

edited May 23 '17 at 11:49

Community

1
1

answered Jun 18 '14 at 12:45

soupault

6,089
4
24
35

Even if the data isn't on disk yet, it should be readable by the other program as soon as the data has reached the OS buffers. – glglgl Jun 18 '14 at 12:56
No doubts. It is written there, where the link leads to, btw. – soupault Jun 18 '14 at 13:04

score 0 · Answer 3 · answered Jun 18 '14 at 12:46

0

You may need to call os.fsync() as well to make sure that the changes are not just flushed but forcefully written to disk as well.

answered Jun 18 '14 at 12:46

Mattias Backman

927
12
25

score 0 · Answer 4 · answered Jun 18 '14 at 13:12

I'm pretty sure that you realise that output is rebound on each iteration, and will therefore contain only the final value at the end, so I've tried a slightly modified version of your code and found that it works as expected.

output = []
with open('test.txt', 'w') as f:
    for i in range(20):
        f.write('%d ' % i)
        f.flush()
        output.append(subprocess.check_output('cat test.txt', shell=True, stderr=subprocess.STDOUT))

>>> from pprint import pprint
>>> pprint(output)
[b'0 ',
 b'0 1 ',
 b'0 1 2 ',
 b'0 1 2 3 ',
 b'0 1 2 3 4 ',
 b'0 1 2 3 4 5 ',
 b'0 1 2 3 4 5 6 ',
 b'0 1 2 3 4 5 6 7 ',
 b'0 1 2 3 4 5 6 7 8 ',
 b'0 1 2 3 4 5 6 7 8 9 ',
 b'0 1 2 3 4 5 6 7 8 9 10 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ',
 b'0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ']

Quick write to a file for a Python subprocess

4 Answers4