Piping data from python to an external command

Question

I've read everything there is on subprocess.Popen but I think I'm missing something.

I need to be able to execute a unix program which reads a data stream from a list created in the python script and write the result of that program to a file. From the bash prompt I do this all the time with no problem but now I am trying to to this from within a python script which preprocesses some binary files and a lot of data before coming to this stage.

Lets look at a simple example not including all the preprocessing:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 .... > outfile'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print process.communicate(str(points))

The way this would be executed in bash is:

echo "11 31
      13 33
      15 37
      16 35
      17 38
      18 39.55" | my_unix_prog option1 option2 .... > outfile

The way the data is fed into the unix prog is important as well, I should be formatted in 2 columns separated by whitespace.

Any help is appreciated...

in the `command` you have the output redirected to `outfile` but then you try to read it into your program using `communicate`. Where do you want the output to go? — Ryan Haining, Aug 06 '12 at 14:16
Using '>' is not the way to redirect output to a file, see this answer: http://stackoverflow.com/questions/8902206/subprocess-popen-io-redirect — Dhara, Aug 06 '12 at 14:20

score 5 · Accepted Answer · edited May 23 '17 at 11:51

5

SOLVED!

With the help of Dhara and xhainingx I was able to figure this out:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 ....'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=open('outfile', 'w'), stderr=PIPE)
for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

This works very well, thanks.

edited May 23 '17 at 11:51

Community

1
1

answered Aug 06 '12 at 15:18

Shahar

886
1
10
22

I'd also recommend that you create the file object outside the call the Popen so you can later call `.close()` on it when you and your subprocess are through with it – Ryan Haining Aug 06 '12 at 18:31

score 1 · Answer 2 · answered Aug 06 '12 at 14:13

1

how about something like

for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

answered Aug 06 '12 at 14:13

Ryan Haining

35,360
15
114
174

Yes, you are correct about that (both of you) but I think my problems begin before that: I use the '>' redirection but this is shell redirection and is not valid. – Shahar Aug 06 '12 at 14:35
@Shahar you can use the `shell=True` argument in your Popen constructor to allow the shell redirection, but then `communicate` won't return anything for stdout. Another option is to open a file `f=open('outfile', 'w')` and then instead of `PIPE` use `stdout=f`. I'm not sure what the problem is now – Ryan Haining Aug 06 '12 at 15:16

score 0 · Answer 3 · answered Aug 06 '12 at 14:16

You need to format your input to communicate correctly.

str will keep the special characters when you print your list of tuples which isn't what you want.

>>> print str([(1,2), (3,4)]) 
[(1,2), (3,4)]

Try this:

print process.communicate("\n".join(["%s %s"%(x[0], x[1]) for x in points])

Piping data from python to an external command

3 Answers3