72

I want to pipe the output of ps -ef to python line by line.

The script I am using is this (first.py) - 

#! /usr/bin/python

import sys

for line in sys.argv:
   print line

Unfortunately, the "line" is split into words separated by whitespace. So, for example, if I do 

echo "days go by and still" | xargs first.py

the output I get is 

./first.py
days
go
by
and
still

How to write the script such that the output is 

./first.py
days go by and still

?

CodeBlue
  • 14,631
  • 33
  • 94
  • 132

4 Answers4

167

Instead of using command line arguments I suggest reading from standard input (stdin). Python has a simple idiom for iterating over lines at stdin:

import sys

for line in sys.stdin:
    sys.stdout.write(line)

My usage example (with above's code saved to iterate-stdin.py):

$ echo -e "first line\nsecond line" | python iterate-stdin.py 
first line
second line

With your example:

$ echo "days go by and still" | python iterate-stdin.py
days go by and still
Dr. Jan-Philip Gehrcke
  • 33,287
  • 14
  • 85
  • 130
  • 1
    Just to clarify: your goal is to read the standard output of one program line by line with your Python program. You are using the command line, you propose use a pipe to transfer the standard output from the first program to your second program (which makes sense). And then, instead of simply reading from standard input in your Python program, you prefer to include a third program to your stack which performs a magic conversion from stdin to commandline arguments and possibly calls your Python program multiple times and fragments the input (do you know how `xargs` works?)? – Dr. Jan-Philip Gehrcke Jul 15 '13 at 16:50
  • 3
    The operating system imposes a limit on the number of commandline arguments that a program can process. `xargs` makes sure that the program defined by the first argument to `xargs` is never called with more args than defined by this limit. It simply calls the program multiple times if required so that all arguments becomed processed. Hence, for large input multiple **independent** runs of your Python program might happen. Generally, commandline arguments are not the place to provide tons of input data. – Dr. Jan-Philip Gehrcke Jul 15 '13 at 17:04
  • Ok. That would be catastrophic. – CodeBlue Jul 15 '13 at 17:05
  • 2
    And how do you check if the user/caller really input something over the pipe or just typed `python ./python_iterate_stdin.py`? – buhtz Nov 23 '15 at 13:19
  • Is it possible to pipe into an interactive Python REPL? Running `echo "days go by and still" | python` or `echo "days go by and still" | python -i` takes the stream as a python script instead of stdin. – anishpatel Dec 08 '16 at 02:03
  • `$ echo "hello" | ./pipe2py.py import: unable to open X server '' @ error/import.c/ImportImageCommand/368. ./pipe2py.py: line 4: syntax error near unexpected token 'sys.stdout.write' ./pipe2py.py: line 4: ' sys.stdout.write(line)'` – abalter Jan 04 '17 at 18:23
  • @abalter you must execute your Python script with Python, and not with your shell :). Do `python pipe2py.py` or add a shebang to the script. – Dr. Jan-Philip Gehrcke Jan 04 '17 at 18:39
  • any reason to `sys.stdout.write` instead of `print`? – Capi Etheriel May 07 '20 at 19:31
  • `ps -ef | python -c "$(echo -e "import sys\nfor line in sys.stdin:\n\tsys.stdout.write(line)")"` – Yzmir Ramirez Jun 01 '20 at 02:27
13

What you want is popen, which makes it possible to directly read the output of a command like you would read a file:

import os
with os.popen('ps -ef') as pse:
    for line in pse:
        print line
        # presumably parse line now

Note that, if you want more complex parsing, you'll have to dig into the documentation of subprocess.Popen.

Vincent Fourmond
  • 3,038
  • 1
  • 22
  • 24
1

Another approach is to use the input() function (the code is for Python 3).

while True:
        try:
            line = input()
            print('The line is:"%s"' % line)
        except EOFError:
            # no more information
            break

The difference between the answer and the answer got by Dr. Jan-Philip Gehrcke is that now each of the lines is without a newline (\n) at the end.

sergzach
  • 6,578
  • 7
  • 46
  • 84
  • i think, thats the most clean answer. Check right away if pipe exist, then read it. If a newline must be there a: .... line+'\n' , can do the job. i upvote like most clean, most minimal code ! – kapad Apr 10 '22 at 21:39
1

I know this is really out-of-date, but you could try

#! /usr/bin/python
import sys
print(sys.argv, len(sys.argv))

if len(sys.argv) == 1:
    message = input()
else:
    message = sys.argv[1:len(sys.argv)]

print('Message:', message)

and I tested it thus:

$ ./test.py
['./test.py'] 1
this is a test
Message: this is a test

$ ./test.py this is a test
['./test.py', 'this', 'is', 'a', 'test'] 5
Message: ['this', 'is', 'a', 'test']

$ ./test.py "this is a test"
['./test.py', 'this is a test'] 2
Message: ['this is a test']

$ ./test.py 'this is a test'
['./test.py', 'this is a test'] 2
Message: ['this is a test']

$ echo "This is a test" | ./test.py
['./test.py'] 1
Message: This is a test

Or, if you wanted the message to be one string, each and every time, then

    message = ' '.join(sys.argv[1:len(sys.argv)])

would do the trick on line 8

spufidoo
  • 11
  • 4