Python subprocess stdout doesn't capture input prompt

Question

I'm using subprocess to spawn a conda create command and capture the resulting stdout for later use. I also immediately print the stdout to the console so the user can still see the progress of the subprocess:

p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
    sys.stdout.write(line.decode(sys.stdout.encoding))

This works fine until half way through the execution when the conda create requires user input: it prompts Proceed (n/y)? and waits for the user to input an option. However, the code above doesn't print the prompt and instead just waits for input "on a blank screen". Once an input is received the prompt is printed afterwards and then execution continues as expected.

I assume this is because the input somehow blocks the prompt being written to stdout and so the readline doesn't receive new data until after the input block has been lifted.

Is there a way I can ensure the input prompt is printed before the subprocess waits for user input? Note I'm running on Windows.

Note that ``readline`` actually looks for an entire line, up to the end-of-line character ``\n``. A ``Proceed (n/y)? y`` prompt does *not* output an entire line but expects input (the ``y``) inline. — MisterMiyagi, Jul 28 '20 at 08:27
You raise a good point. Is it likely a simple fix using `read` instead? — Mrfence, Jul 28 '20 at 11:58

score 4 · Accepted Answer · answered Jul 28 '20 at 15:34

Although I'm sure pexpect would have worked in this case, I decided it would be overkill. Instead I used MisterMiyagi's insight and replaced readline with read.

The final code is as so:

p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while p.poll() is None:
    sys.stdout.write(p.stdout.read(1).decode(sys.stdout.encoding))
    sys.stdout.flush()

Note the read(1) as just using read() would block the while loop until an EOF is found. Given no EOF is found before the input prompt, nothing will be printed at all! In addition, flush is called which ensures the text written to sys.stdout is actually visible on screen.

proc.stdout.isatty() gives False, and proc.stdout.writable() as well. Why is it that much complicated ? proc.stdout.read(1) get stuck in a infinite loop. — Thomas Aumaitre, Apr 25 '23 at 05:08

score 0 · Answer 2 · answered Jul 28 '20 at 08:26

For this use case I recommend using pexpect. stdin != stdout

Example use case where it conditionally sends to stdin on prompts on stdout

def expectgit(alog):
    TMPLOG = "/tmp/pexpect.log"
    cmd = f'''
ssh -T git@github.com ;\
echo "alldone" ;
'''
    with open(TMPLOG, "w") as log:
        ch = pexpect.spawn(f"/bin/bash -c \"{cmd}\"", encoding='utf-8', logfile=log)
        while True:
            i = ch.expect(["successfully authenticated", "Are you sure you want to continue connecting"])
            if i == 0:
                alog.info("Git all good")
                break
            elif i == 1:
                alog.info("Fingerprint verification")
                ch.send("yes\r")
        ch.expect("alldone")
        i = ch.expect([pexpect.EOF], timeout=5)
        ch.close()
        alog.info("expect done - EOF")

    with open(TMPLOG) as log:
        for l in log.readlines():
            alog.debug(l.strip())

This works pretty slick for getting the output to the screen. How to accomplish capturing the user response for sending back to the subprocess...? — Robatron, Sep 08 '21 at 22:02

Python subprocess stdout doesn't capture input prompt

2 Answers2

Linked