2

I'm writing a python script that runs a command using subprocess module and then writes the output to a file. Since the output is too big, I want to write just x last lines of the output that contains the wanted information.

import subprocess
outFile=open('output.txt', 'w')
proc=subprocess.Popen(command, cwd=rundir, stdout=outFile)

The above code writes the whole output to a file (which is very very large), but what I want is just an x number of lines from the end of the output.

EDIT: I know that I can post-process the file afterward, but what I want really is to write just the lines I need from the beginning without handling all those data.

Youcef4k
  • 338
  • 2
  • 13

2 Answers2

1

I would suggest to store the output in a variable and then do some processing. The Python Interpreter will take care of all the data that is produced - even if it is larger than your RAM.

CODE

with open('output.txt', 'w') as fobj:
    out = subprocess.check_output(command).rstrip().splitlines()
    fobj.write('\n'.join(out[-MAX_LINES:]))

EXPLANATION

  • The function subprocess.check_output returns the console output as string.
  • The method str.rstrip() returns the the string lack of trailing whitespaces. So the parameter MAX_LINES has control of the last non-empty lines.
  • The method str.splitlines() returns a list of strings, each represents one line.
  • out[-MAX_LINES:]
    • When MAX_LINES > len(out), this will return the whole output as list.

COMMENT

Always use context managers (with ...)!!! This is more safe for file management.

Sven-Eric Krüger
  • 1,277
  • 12
  • 19
  • When he says that the file is "very very large", that is because he does not want to process it fully, but your solution puts it into memory which I thinh is not doable for him. – Lenormju May 07 '19 at 12:55
  • @Lenormju The Python Interpreter will do the steps necessary process all the data. – Sven-Eric Krüger May 07 '19 at 12:58
0

You can either truncate the file after it has been fully written, or give an io.StringIO to your process, which you can getvalue() and then write only the lines you want.

Lenormju
  • 4,078
  • 2
  • 8
  • 22
  • That is not different from post-processing the written file, which I want to avoid, is there a way to specify a part of the output in the stdout option of subprocess? – Youcef4k May 07 '19 at 12:40
  • According to the documentation (https://docs.python.org/3.5/library/subprocess.html#popen-constructor), there is no way to do it there, maybe check this answer (https://stackoverflow.com/a/42173309/11384184) but is looks like there is no easy solution. – Lenormju May 07 '19 at 12:52