Read / Write simultaneously python subprocess.Popen

Question

I have a simple C program which works the following way: Ask for input Print it Ask another input Print again

Now iam using python to call this program.

import subprocess

sobj = subprocess.Popen("./cprog", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
sobj.stdin.write("2 3\n")
sobj.stdin.close()
sobj.stdout.read()

This works fine. Similarly with communicate its working fine.

But when I try to do something like this it won't work

sobj = subprocess.Popen("./cprog", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
sobj.stdout.readline()
sobj.stdin.write("2 3\n")
sobj.stdin.close()
sobj.stdout.read()

Here are the few things: 1. I saw pexpect but I think we should give what program asks in advance. 2. Can I reopen closed subprocess pipe ?

Iam using the above script as CGI and I don't know why but subprocess.call won't work in that. Can anyone explain why?

EDIT:

Iam doing a web based project where users write code in either C, C++ or JAVA and execute them on browser. So first I thought of using PHP for it but I couldn't find a way to call programs and run them interactively. Then I saw python subprocess module. Everything was working fine in interpreter when I was using subprocess.call. But the same python program when saved it as .cgi and opened it in browser it didn't work. Then I started looking at subprocess.popen. But with this I need to give all the inputs in beginning and then run the code. What I want to do is run an interactive session in browser.

EDIT 2: So what I want is user runs program in browser and enters input in textbox provided whenever needed and that input is redirected to stdin of subprocess and output based on it.

EDIT 3: cprog.c

 #include <stdio.h>
 int main() {
   int x;
   printf("Enter value of x: \n");
   scanf("%d", &x);
   printf("Value of x: %d\n", x);
   return 0;
 }

Answer 1

I'm assuming your C application displays a prompt and expects the user to enter their input on the same line, and that in your readline() call above you're trying to get the prompt.

If this is the case, readline() will block forever because it's waiting for a newline character and never seeing it. If you convert this call to a simple read(X) (where X is a number of bytes to read in one go) then you'll probably have better luck, although you should cope with partial input (i.e. loop around collecting input until you've seen the whole prompt). The only other issue you might see is if the C application isn't flushing the output before prompting the user, but I'd expect you to see that problem in the interactive session as well if that were the case.

When running under the context of a webserver like Apache then it's generally a bad idea to use things like subprocess as they involve forking additional processes and that's often quite a tricky thing to manage. This is because the fork process duplicates much of the state of the parent and sometimes this can cause issues. I'm not saying it won't work, I'm just saying you can make some subtle problems for yourself if you're not careful, and it wouldn't surprise me if that's why you're having trouble using subprocess.

To give any more helpful advice, though, you'd need to describe exactly the error you see when you call subprocess. For example, there's quite likely an exception being thrown which will probably be in your webserver logs - reproducing that here would be a good start.

Answer 2

When I run C program directly through terminal its working fine. But when I run the same program with 2nd code i provided above nothing prints.

The reason you don't see any output is that C stdio uses block-buffering when the program is run in non-interactive mode. See my answer that demonstrate several solutions: pty, stdbuf, pexpect. If you can change the C code then you could also fflush the output explicitly or make it unbuffered.

If you can provide all input at once and the output is bounded then you could use .communicate():

from subprocess import Popen, PIPE

p = Popen(["./cprog"], stdin=PIPE, stdout=PIPE, stderr=PIPE,
          universal_newlines=True)
out, err = p.communicate("2\n")

---

So what I want is user runs program in browser and enters input in textbox provided whenever needed and that input is redirected to stdin of subprocess and output based on it.

Based on ws-cli example:

#!/usr/bin/python
"""WebSocket CLI interface.

Install: pip install twisted txws

Run: twistd -ny wscli.py

Visit http://localhost:8080/
"""
import sys
from twisted.application import strports # pip install twisted
from twisted.application import service
from twisted.internet    import protocol
from twisted.python      import log
from twisted.web.resource import Resource
from twisted.web.server  import Site
from twisted.web.static  import File

from txws import WebSocketFactory # pip install txws


class Protocol(protocol.Protocol):
    def connectionMade(self):
        from twisted.internet import reactor
        log.msg("launch a new process on each new connection")
        self.pp = ProcessProtocol()
        self.pp.factory = self
        reactor.spawnProcess(self.pp, command, command_args)
    def dataReceived(self, data):
        log.msg("redirect received data to process' stdin: %r" % data)
        self.pp.transport.write(data)
    def connectionLost(self, reason):
        self.pp.transport.loseConnection()

    def _send(self, data):
        self.transport.write(data) # send back


class ProcessProtocol(protocol.ProcessProtocol):
    def connectionMade(self):
        log.msg("connectionMade")
    def outReceived(self, data):
        log.msg("send stdout back %r" % data)
        self._sendback(data)
    def errReceived(self, data):
        log.msg("send stderr back %r" % data)
        self._sendback(data)
    def processExited(self, reason):
        log.msg("processExited")
        self._sendback('program exited')
    def processEnded(self, reason):
        log.msg("processEnded")

    def _sendback(self, data):
        self.factory._send(data)

command = './cprog'
command_args = [command]

application = service.Application("ws-cli")

echofactory = protocol.Factory()
echofactory.protocol = Protocol
strports.service("tcp:8076:interface=127.0.0.1",
                 WebSocketFactory(echofactory)).setServiceParent(application)

resource = Resource()
resource.putChild('', File('index.html'))
strports.service("tcp:8080:interface=127.0.0.1",
                 Site(resource)).setServiceParent(application)

where index.html:

<!doctype html>
<title>Send input to subprocess using websocket and echo the response</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js">
</script>
<script>
// send keys to websocket and echo the response
$(document).ready(function() {
    // create websocket
    if (! ("WebSocket" in window)) WebSocket = MozWebSocket; // firefox
    var socket = new WebSocket("ws://localhost:8076");

    // open the socket
    socket.onopen = function(event) {
    // show server response
    socket.onmessage = function(e) {
        $("#output").text(e.data);
    }

    // sent input
    $("#entry").keyup(function (e) {
        socket.send($("#entry").attr("value")+"\n");
    });
    }
});
</script>
<pre id=output>Here you should see the output from the command</pre>
<input type=text id=entry value="123">

And cprog.c:

#include <stdio.h>
int main() {
  int x = -1;
  setbuf(stdout, NULL); // make stdout unbuffered

  while (1) {
    printf("Enter value of x: \n");
    if (scanf("%d", &x) != 1)
        return 1;
    printf("Value of x: %d\n", x);
  }
  return 0;
}