`stdout` is empty when passing data to `stdin` via `subprocess`, but contains the expected output when input is entered manually?

Question

So I'm working on trying to receive chess move data from the Leela Chess Zero engine. I am already done with all of the UI and other parts of the backend, and this is the last thing I need to implement. Unfortunately, I seem to have overestimated how simple subprocessing in Python is...

To explain a little more background, all I need to do is:

1. Call/run lc0.exe in the local directory. I have managed this just fine.
2. Pass the commands position startpos e2e4 e7e5 e1e2, go nodes 100, quit via stdin in that order. This also seems to work fine according to what I can gauge via stderr.
3. Receive/read stdout. This is where I'm stuck.

Here's the things I've tried so far:

>>> from subprocess import Popen, PIPE, STDOUT
>>> p = Popen(['lc0'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
>>> stdout_data = p.communicate(input=b'position startpos e2e4 e7e5 e1e2\ngo nodes 100\nquit')[0]
>>> stdout_data
b''

I get an empty byte string. I then tried a different method as a test:

>>> import subprocess
>>> subprocess.check_output(['lc0'], stderr=PIPE) #the following 3 lines is me typing into stdin
position startpos e2e4 e7e5 e1e2
go nodes 100
quit
b'info depth 1 seldepth 2 time 4081 nodes 4 score cp 12 nps 133 tbhits 0 pv e2e4 c7c5\r\ninfo depth 2 seldepth 3 time 4116 nodes 11 score cp 13 nps 166 tbhits 0 pv e2e4 c7c5 g1f3\r\ninfo depth 3 seldepth 4 time 4151 nodes 25 score cp 13 nps 247 tbhits 0 pv e2e4 c7c5 g1f3 e7e6\r\ninfo depth 3 seldepth 5 time 4218 nodes 68 score cp 13 nps 407 tbhits 0 pv e2e4 c7c5 b1c3 b8c6 g1e2\r\ninfo depth 4 seldepth 6 time 4312 nodes 134 score cp 13 nps 513 tbhits 0 pv e2e4 c7c5 b1c3 b8c6 g1f3 e7e5\r\nbestmove e2e4 ponder c7c5\r\n'

Eureka! I received the correct output from stdout. Now time to do it programatically:

>>> subprocess.check_output(['lc0'], stderr=PIPE, input=b'position startpos e2e4 e7e5 e1e2\ngo nodes 100\nquit')
b''

Bugger! What is happening here? I can confirm by removing the stderr=PIPE argument that all of the commands apparently are indeed being run by the engine, but when all is said and done, stdout is empty when I pass the commands programatically. I've also tried using subprocess.stdin.write() with identical results.

After digging a lot, I found that pexpect might be more suitable for this use case. I installed pip install wexpect and lo' and behold:

>>> from wexpect import spawn
>>> child = spawn("lc0")

...

Yup, it just hangs. Breaking with ^C gives me the exception pywintypes.error: (2, 'CreateFile', 'The system cannot find the file specified.'), so I understandably feel less confident about using pexpect instead of subprocess, since I seem much closer to a solution with the latter.

Anyways, I'm convinced I'm misuing subprocess somehow, but what exactly am I doing wrong? Why am I correctly receiving stdout when passing commands through stdin manually, but not when using the input= argument?

Answer 1

We can use stdin.write() to send command to the engine one at a time. Also properly send the position command with moves as in: position startpos moves m1, m2 ... Safely quit the engine and properly terminate the process.

Code

from subprocess import Popen, PIPE, STDOUT, TimeoutExpired


def ecommand(p, comm):
    p.stdin.write(f'{comm}\n')


def analyze(efile):
    bestmove = '0000'

    p = Popen([efile], stdout=PIPE, stdin=PIPE, stderr=STDOUT, bufsize=0, text=True)  # stderr=STDOUT, also send stderr to stdout to see everything in stdout

    ecommand(p, 'position startpos moves e2e4 e7e5 e1e2')
    ecommand(p, 'go nodes 3000')

    for line in iter(p.stdout.readline, ''):  # read each line of engine output as replies from our command
        line = line.strip()
        print(line)

        if line.startswith('bestmove'):  # exit the loop when we get the engine bestmove
            bestmove = line.split()[1].strip()
            break

    ecommand(p, 'quit')  # properly quit the engine

    # Make sure process 'p' is terminated (if not terminated for some reason) as we already sent the quit command.
    try:
        p.communicate(timeout=5)
    except TimeoutExpired:  # If timeout has expired and process is still not terminated.
        p.kill()
        p.communicate()

    return bestmove


efile = 'E:\\Chess_Engines\\Lc0\\lc0-v0.28.0-windows-cpu-openblas\\lc0.exe'
bestmove = analyze(efile)
print(f'best move: {bestmove}')

Output

_
|   _ | |
|_ |_ |_| v0.28.0 built Aug 25 2021
Detected 4 core(s) and 8 thread(s) in 1 group(s).
...

info depth 1 seldepth 2 time 181 nodes 2 score cp 88 nps 142 tbhits 0 pv g8f6 d2d3
info depth 2 seldepth 3 time 200 nodes 3 score cp 89 nps 90 tbhits 0 pv g8f6 d2d3 d7d5
info depth 2 seldepth 4 time 236 nodes 5 score cp 93 nps 72 tbhits 0 pv g8f6 b1c3 b8c6 d2d3
info depth 3 seldepth 5 time 305 nodes 15 score cp 99 nps 107 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 d5d4
info depth 3 seldepth 6 time 368 nodes 23 score cp 96 nps 114 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 d5d4 c3b1
info depth 4 seldepth 7 time 466 nodes 34 score cp 105 nps 113 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 d5d4 c3b1 b8c6
info depth 4 seldepth 8 time 561 nodes 55 score cp 106 nps 139 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 d5d4 c3b1 b8c6
info depth 5 seldepth 8 time 871 nodes 105 score cp 109 nps 149 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3
info depth 5 seldepth 9 time 965 nodes 119 score cp 111 nps 149 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3
info depth 5 seldepth 10 time 1327 nodes 179 score cp 109 nps 154 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3 d5d4
info depth 5 seldepth 11 time 1444 nodes 206 score cp 110 nps 161 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3 d5d4 c3b1
info depth 5 seldepth 12 time 1621 nodes 240 score cp 110 nps 165 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3 d5d4 c3b1
info depth 6 seldepth 12 time 2126 nodes 331 score cp 112 nps 168 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3 d5d4 c3b1
info depth 6 seldepth 13 time 2582 nodes 430 score cp 111 nps 178 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 g1f3 d5d4 c3b1
info depth 6 seldepth 14 time 5050 nodes 1037 score cp 111 nps 212 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 e4d5 c6d4 e2d2 e6d5
info depth 7 seldepth 14 time 5569 nodes 1161 score cp 111 nps 214 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c8e6 e4d5 c6d4 e2d2 e6d5
info depth 7 seldepth 15 time 8385 nodes 1938 score cp 109 nps 235 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c6d4 e2e1 c7c6 g1f3 d8b6 a1b1 c8e6 f3e5
info depth 7 seldepth 15 time 8642 nodes 2007 score cp 109 nps 236 tbhits 0 pv g8f6 d2d3 d7d5 b1c3 b8c6 c1g5 c6d4 e2e1 c7c6 g1f3 d8b6 a1b1 c8e6 f3e5
bestmove g8f6 ponder d2d3
best move: g8f6