Python subprocess output blocking on application prompt

Question

I am running jirashell in a python script using the subprocess library. I am currently having issues having the outputs print in real time. When I run jirashell it outputs information than prompts the user (y/n). The subprocess won't print out information prior to the prompt until I enter 'y' or 'n'.

The code I am using is

_consumer_key = "justin-git"
_cmd = "jirashell -s {0} -od -k {1} -ck {2} -pt".format(JIRA_SERVER,
    _rsa_private_key_path, _consumer_key)
p = subprocess.Popen(_cmd.split(" "), stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE, bufsize=0)

out, err = p.communicate() # Blocks here

print out
print err

The output is like so:

n # I enter a "n" before program will print output.
Output:
Request tokens received.
    Request token:        asqvavefafegagadggsgewgqqegqgqge
    Request token secret: asdbresbdfbrebsaerbbsbdabweabfbb
Please visit this URL to authorize the OAuth request:
        http://localhost:8000/plugins/servlet/oauth/authorize?oauth_token=zzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Have you authorized this program to connect on your behalf to http://localhost:8000? (y/n)
Error:
Abandoning OAuth dance. Your partner faceplants. The audience boos. You feel shame.

Does anyone know how I can have it print the output prior to the prompt than wait for an input of y/n? Note I also need to be able to store the output produced by the command so "os.system()" won't work...

EDIT:

It looks like inside jirashell there is a part of the code that is waiting for an input and this is causing the block. Until something is passed into this input nothing is outputted... Still looking into how I can get around this. I'm in the process of trying to move the portion of code I need into my application. This solution doesn't seem elegant but I can't see any other way right now.

approved = input(
    'Have you authorized this program to connect on your behalf to {}? (y/n)'.format(server))

Answer 1

Method which prints and caches the standard output:

You can use a thread which reads the standard output of your subprocess, while the main thread is blocked until the subprocess is done. The following example will run the program other.py, which looks like

#!/usr/bin/env python3

print("Hello")
x = input("Type 'yes': ")

Example:

import threading
import subprocess
import sys

class LivePrinter(threading.Thread):
    """
    Thread which reads byte-by-byte from the input stream and writes it to the
    standard out. 
    """
    def __init__(self, stream):
        self.stream = stream
        self.log = bytearray()
        super().__init__()

    def run(self):
        while True:
            # read one byte from the stream
            buf = self.stream.read(1)

            # break if end of file reached
            if len(buf) == 0:
                break

            # save output to internal log
            self.log.extend(buf)

            # write and flush to main standard output
            sys.stdout.buffer.write(buf)
            sys.stdout.flush()

# create subprocess
p = subprocess.Popen('./other.py', stdout=subprocess.PIPE)

# Create reader and start the thread
r = LivePrinter(p.stdout)
r.start()

# Wait until subprocess is done
p.wait()

# Print summary
print(" -- The process is done now -- ")
print("The standard output was:")
print(r.log.decode("utf-8"))

The class LivePrinter reads every byte from the subprocess and writes it to the standard output. (I have to admit, this is not the most efficient approach, but a larger buffer size blocks, the LiveReader until the buffer is full, even though the subprocess is awaiting the answer to a prompt.) Since the bytes are written to sys.stdout.buffer, there shouldn't be a problem with multi-byte utf-8 characters.

The LiveReader class also stores the complete output of the subprocess in the variable log for later use.

As this answer summarizes, it is save to start a thread after forking with subprocess.

---

Original answer which has problems, when the prompt line doesn't end a line:

The output is delayed because communicate() blocks the execution of your script until the sub-process is done (https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate).

You can read and print the standard output of the subprocess, while it is executed using stdout.readline. There are some issues about buffering, which require this rather complicated iter(process.stdout.readline, b'') construct. The following example uses gpg2 --gen-key because this command starts an interactive tool.

import subprocess

process = subprocess.Popen(["gpg2", "--gen-key"], stdout=subprocess.PIPE)

for stdout_line in iter(process.stdout.readline, b''):
    print(stdout_line.rstrip())

---

Alternative answer which uses shell and does not cache the output:

As Sam pointed out, there is a problem with the above solution, when the prompt line does not end the line (which prompts they usually don't). An alternative solution is to use the shell argument to interact with the sub-process.

import subprocess
subprocess.call("gpg2 --gen-key", shell=True)