4

I found a question on this site which showed me how to call a Perl script from Python. I'm currently using the following lines of code to achieve this:

pipe = subprocess.Popen(["perl", "./Perl_Script.pl", param], stdout=subprocess.PIPE)
result = pipe.stdout.read()

This works perfectly, but the only issue is that the Perl script takes a few minutes to run. At the end of the Perl script, I use a simple print statement to print my values I need to return back to Python, which gets set to the result variable in Python.

Is there a way I can include more print statements in my Perl script every few seconds that can get returned to Python continuously (instead of waiting a few minutes and returning a long list at the end)?

Ultimately, what I'm doing is using the Perl script to obtain data points that I then send back to Python to plot an eye diagram. Instead of waiting for minutes to plot the eye diagram when the Perl script is finished running, I'd like to return segments of the data to Python continuously, allowing my plot to update every few seconds.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
dmranck
  • 131
  • 1
  • 2
  • 9
  • Have you tried simply printing at intermediate stages in the perl script? I don't see what the question is. – Brenden Brown Nov 29 '12 at 21:08
  • Yes, but the python variable _result_ is not updated until the perl script has finished running. And yes, I can put more print statements throughout the perl script and these do get returned - but they get returned all together when the perl script is done. – dmranck Nov 29 '12 at 21:12
  • There is an answer below talking about turning off buffering when pushing to the pipe, and another about making sure that you're not buffering pipe input. You need to do both. – Jeff Ferland Nov 29 '12 at 21:33
  • @JeffFerland on my machine, python's input is line-buffered. I'm not sure about perl, though – loopbackbee Nov 29 '12 at 21:44

3 Answers3

2

The default UNIX stdio buffer is at least 8k. If you're writing less than 8k, you'll end up waiting until the program ends before the buffer is flushed.

Tell the Perl program to stop buffering output, and probably tell python not to buffer input through the pipe.

$| = 1;

to un-buffer STDOUT in your Perl program.

Len Jaffe
  • 3,442
  • 1
  • 21
  • 28
  • Aren't most unix tools (that handle text) line-buffered by default? `cat -` displays that behaviour (each line is immediately flushed). – loopbackbee Nov 29 '12 at 21:31
  • 1
    If you use the read() and write() system calls directly, you do un-buffered output. if you use the stdio library, you're using buffered IO, because stdio implements the buffering on top of read/write. If you want to use the stdio function, but without buffering, you have to use a function call to set the size of the buffer to 0. I don't know how cat is implemented, but stdio isn't like buffered, but does offer several line-oriented functions. But a line is defined as a bunch of text until the next occurrence of the line delimiter (which defaults to \n on UNIX and \r\n on windows). – Len Jaffe Nov 29 '12 at 21:47
  • Perl's STDOUT to a pipe is block-buffered by default. See `$OUTPUT_AUTOFLUSH` in http://perldoc.perl.org/perlvar.html . – darch Nov 29 '12 at 22:36
  • Yes. That's what I said. $OUTPUT_AUTOFLUSH is an alias for $| if you 'use English'. – Len Jaffe Nov 29 '12 at 22:52
1

pipe.stdout.read() tries to read the whole stream, so it will block until perl is finished. Try this:

line=' '
while line:
    line = pipe.stdout.readline()
    print line,
loopbackbee
  • 21,962
  • 10
  • 62
  • 97
1

You need two pieces: To read a line at a time in Python space and to emit a line at a time from Perl. The first can be accomplished with a loop like

while True:
  result = pipe.stdout.readline()
  if not result:
    break
  # do something with result

The readline blocks until a line of text (or EOF) is received from the attached process, then gives you the data it read. So long as each chunk of data is on its own line, that should work.

If you run this code without modifying the Perl script, however, you will not get any output for quite a while, possibly until the Perl script is finished executing. This is because Perl block-buffers output to a pipe by default. You can tell it to flush the buffer more often by changing a global variable in the scope in which you are printing:

use English qw(-no_match_vars);
local $OUTPUT_AUTOFLUSH = 1;
print ...;

See http://perl.plover.com/FAQs/Buffering.html and http://perldoc.perl.org/perlvar.html .

darch
  • 4,200
  • 1
  • 20
  • 23