1

Is there a best-practices approach to poll for the stdout/stderr from a subprocess.Popen as well as a zmq socket?

In my case, I have my main program spawning a Popen subprocess. The subprocess publishes messages via zmq which I then want to subscribe to in my main program.

Waiting on multiple zmq sockets is not complicated with the zmq.Poller but when I want to interleave this with the output from my subprocess itself, I am unsure how to do it in the best way without risking waits or having needless loops.

In the end, I would like to use it like so:

process = Popen([prog, '--publish-to', 'tcp://127.0.0.1:89890'],
        stdout=subprocess.PIPE, stderr=subprocess.PIPE, ...)
for (origin, data) in interleave(process, 'tcp://127.0.0.1:89890'):
    if origin == 'STDOUT': pass
    if origin == 'STDERR': pass
    if origin == 'ZMQ': pass

prog --publish-to tcp://127.0.0.1:89890 will then open a zmq.PUB socket and publish data, whereas the interleave function will subscribe to this and also poll for stdout and stderr, yielding whatever data reaches it first.

I know how to define interleave with multiple daemon threads and queues but I don’t know if this approach might have some caveats with regards to lazy reading (ie. stdout might not be processed until the end of the program?) or other things that I have not yet thought about (seems also to be quite a bit of overhead for such a task).

I will be thankful for all ideas or insights.

I aim for at least Python 3.3/3.4 but if this turns out to be much easier with the new async/await tools, I could also use Python 3.5 for the code.

Debilski
  • 66,976
  • 12
  • 110
  • 133
  • You can get the integer file descriptor using `process.stdout.fileno()`. If you can get the file descriptor of the zmq socket, you can do a `select.select((fd1, d2, fd3), (), (), None)` call to wait for the first one. If there is no way to get the file descriptor of the zmq socket, you can create 2 or 3 threads, both of them reading (and being blocked on) from one source, putting read data to a `Queue`. – pts Jul 13 '16 at 16:02
  • Hmm, will `select` work on windows as well? That would be a stopper, unfortunately. – Debilski Jul 13 '16 at 16:30
  • See also: http://stackoverflow.com/questions/9448247/zeromq-zmq-poller-stdin – Debilski Jul 14 '16 at 06:51

1 Answers1

1

Use zmq.Poller: http://pyzmq.readthedocs.io/en/latest/api/zmq.html#polling. You can register zmq sockets and native file descriptors (e.g. process.stdout.fileno() and process.stderr.fileno()) there, and it will wait until input is available on at least one of the registered sources.

I don't know if it works in Windows, you should try.

pts
  • 80,836
  • 20
  • 110
  • 183
  • Should not work on Windows, unfortunately, but I found a workaround at least. Thanks. http://stackoverflow.com/a/20807648/200266 – Debilski Jul 14 '16 at 06:53