error checking pipe commands made in python for unix

Question

I am compiling a command to be executed on a unix system from python, containing multiple piping step. e.g.:

grep foo myfile | cmd1 | cmd2 > output

These correspond to a series of transformations on entries that have foo in them from myfile. I sometimes construct it as a command to be executed by os.system and in other cases using the subprocess module.

How can I do error checking on each part of the pipe, from Python? For example in 99% of cases, there are foo entries in myfile and the remaining pipe works. But if for some reason myfile exists but contains no foo entries, then the rest of the pipe breaks because you are piping empty files, and the remaining commands require non-empty inputs to work. This makes it hard to debug because all you see is the output of a broken pipe, not knowing which intermediate step failed.

Is there a way to construct pipes in Python that error check intermediate steps? E.g. check that the grep portion of the pipe really contains some output, that the cmd1 output on grep's input also contained some output, and so on? thanks.

The easiest way to do this is to do things natively in Python. Calling back and forth from the OS and Python just introduces extra variables and complexity that are usually best avoided in the first place. — Silas Ray, Dec 17 '12 at 18:50
So how would you do it in Python if it requires calling an external command? The only way I know is ``subprocess``. I cannot rewrite all of the commands I call in Python — , Dec 17 '12 at 19:36
In bash you have the PIPESTATUS variable -- use that if you can. Otherwise just do it in python like sr2222 suggested -- setup the pipes, launch the programs with redirected input output, check their exit status, what's the problem with that? The module you mentioned even takes stdin/stdout/stderr as parameters! — loreb, Dec 17 '12 at 20:11

score 1 · Answer 1 · answered Jan 08 '14 at 09:19

Here is my approach, tested on Python 2.6 Linux.

a.Define a method

def run_command(command):
    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out, err = p.communicate()
    return (p.returncode, out, err)

b.Use it in the following way

def test_run_command_pipestatus(self):
    # the exit $PIPESTATUS is the return code of "exit 2". The default return code is the return code of last command in the pipe
    return_code, out, err = run_command("exit 2 | tail -n 1; exit $PIPESTATUS")
    print "return_code = %d" %return_code
    print out
    print err
    self.assertEqual(2, return_code)

score 0 · Answer 2 · answered Nov 26 '16 at 20:06

Basically the same as Mingjiang Shi's answer, which has mostly all the key info you need, but without the extra function, and fixing an issue. (and tested in python 2.7.12 and 3.5.2)

$PIPESTATUS doesn't exist in sh, so you must set executable to bash if your distro doesn't have a /bin/sh that acts like bash (such as Ubuntu which uses dash). And it's an array, so I prefer to use it like one.

And I recommend wait() instead of communicate(), so you can filter instead of slurp (use as you read into a small buffer instead of load whole output in memory before processing any).

So all you need to do is:

should use a string, not a list for the args
use shell=True
use executable="/bin/bash"

code:

p = subprocess.Popen("false | true; exit ${PIPESTATUS[0]}", shell=True, executable='/bin/bash', stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# at this point you do your filtering, eg. using p.stdout.readinto(buf)

p.wait()
if p.returncode != 0:
    # handle it here

uitb · Answer 3 · 2020-11-20T07:37:38.117

this is my approach, no mater how mutch pipe commands. it sum of all the state codes, and exit with the result "echo 1 |echo 2" or "cat test |echo 2" replace by your commands

>>getstatusoutput('status=0;echo 1 |echo 2;for ((i=0;i<${#PIPESTATUS[*]};i++));do let status=${status}+${PIPESTATUS[${i}]};done;exit $status')

(0, '2')

>> getstatusoutput('status=0;cat test |echo 2;for ((i=0;i<${#PIPESTATUS[*]};i++));do let status=${status}+${PIPESTATUS[${i}]};done;exit $status')

(1, '2\ncat: test: No such file or directory')

error checking pipe commands made in python for unix

3 Answers3