nodejs/express - stream stdout instantly to the client

Question

I spawned the following child: var spw = spawn('ping', ['-n','10', '127.0.0.1']) and I would like to receive the ping results on the client side (browser) one by one, not as a whole.

So far I tried this:

app.get('/path', function(req, res) {
   ...
   spw.stdout.on('data', function (data) {
      var str = data.toString();
      res.write(str + "\n");
   });
   ...
}

and that:

...
spw.stdout.pipe(res);
...

In both cases browser waits 10 of the pings to complete, and then prints the result as a whole. I would like to have them one by one, how to accomplish that?

(Client is just making a call to .../path and console.logs the result)

EDIT: Although I do believe that websockets are necessary to implement this, I just want to know whether there are any other ways. I saw several confusing SO answers, and blog posts (in this post, at step one OP streams the logs to the browser) which didn't help, therefore I decided to go for a bounty for some attention.

score 41 · Accepted Answer · answered Feb 23 '14 at 21:11

Here's a complete example using SSE (Server sent events). This works in Firefox and probably Chrome too:

var cp = require("child_process"),
         express = require("express"),
         app = express();

app.configure(function(){
    app.use(express.static(__dirname));
});


app.get('/msg', function(req, res){
    res.writeHead(200, { "Content-Type": "text/event-stream",
                         "Cache-control": "no-cache" });

    var spw = cp.spawn('ping', ['-c', '100', '127.0.0.1']),
    str = "";

    spw.stdout.on('data', function (data) {
        str += data.toString();

        // just so we can see the server is doing something
        console.log("data");

        // Flush out line by line.
        var lines = str.split("\n");
        for(var i in lines) {
            if(i == lines.length - 1) {
                str = lines[i];
            } else{
                // Note: The double-newline is *required*
                res.write('data: ' + lines[i] + "\n\n");
            }
        }
    });

    spw.on('close', function (code) {
        res.end(str);
    });

    spw.stderr.on('data', function (data) {
        res.end('stderr: ' + data);
    });
});

app.listen(4000);

And the client HTML:

<!DOCTYPE Html>
<html> 
<body>
   <ul id="eventlist"> </ul>

   <script>              
    var eventList = document.getElementById("eventlist");
    var evtSource = new EventSource("http://localhost:4000/msg");

    var newElement = document.createElement("li");
    newElement.innerHTML = "Messages:";
    eventList.appendChild(newElement);


    evtSource.onmessage = function(e) {
        console.log("received event");
        console.log(e);
        var newElement = document.createElement("li");

        newElement.innerHTML = "message: " + e.data;
        eventList.appendChild(newElement);
    };      

    evtSource.onerror = function(e) {
        console.log("EventSource failed.");
    };

    console.log(evtSource);

    </script>

</body>
</html>

Run node index.js and point your browser at http://localhost:4000/client.html. Note that I had to use the "-c" option rather than "-n" since I'm running OS X.

exactly what I am looking for, will give your bounty once it allows me :) http://imgur.com/oLLbGXU — anvarik, Feb 24 '14 at 11:36
what's a good solution to cancel the event stream when the user navigate from the page? — Ahmed Ahmed, Apr 10 '17 at 20:18
@AhmedAhmed I would assume that if the user had closed the webpage either res.write() would throw an exception or the 'close' event would be triggered. Why don't you try it yourself and report your results here? — nimrodm, Apr 11 '17 at 06:40
I did try it actually :) from the client you need to call evtSource.close(); and from the server you need to listen to the close event req.connection.addListener("close", function () { console.log('client closed'); spw.stdin.pause(); spw.kill(); }); — Ahmed Ahmed, Apr 11 '17 at 14:35
7 years later it still works , but `app.configured` should be removed (just `app.use(express.static(__dirname));` is enough) — Shrike, Apr 06 '21 at 13:00
Great answer. Just two notes: 1) `app.configure` was removed from `express` v4 (see Ahmeds comment above). 2) The request was, for whatever reason, blocked by CORS (which was solved simply by installing `cors` module and adding `app.use(cors())` line. — HynekS, Jan 10 '22 at 06:59
This has worked well for me, with with one issue: The calls to `res.end` are being received by my browsers as an error event, which in turn is causing the EventSource request to start over unless explicitly handled and closed. — JamesP, Apr 13 '22 at 15:58

score 3 · Answer 2 · answered Feb 21 '14 at 18:01

3

If you are using Google Chrome, changing the content-type to "text/event-stream" does what your looking for.

res.writeHead(200, { "Content-Type": "text/event-stream" });

See my gist for complete example: https://gist.github.com/sfarthin/9139500

answered Feb 21 '14 at 18:01

Steve Farthing

792
6
10

Can you try your gist with var spw = cp.spawn('ping', ['127.0.0.1', '-n', '10']), and confirm that you can see real time console on the browser as well? Upon @jibsales answer I started to think that real time browser is only possible with sockets, but your answer confused me... – anvarik Feb 22 '14 at 23:47
just tried it, does not work. if you send 10 pings, your solution waits till all complete, then chrome displays all... – anvarik Feb 23 '14 at 09:51
@anvarik: See the code in my answer. It's based on Steve's suggestion and does work. – nimrodm Feb 24 '14 at 06:42

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

1

This cannot be achieved with the standard HTTP request/response cycle. Basically what you are trying to do is make a "push" or "realtime" server. This can only be achieved with xhr-polling or websockets.

Code Example 1:

app.get('/path', function(req, res) {
   ...
   spw.stdout.on('data', function (data) {
      var str = data.toString();
      res.write(str + "\n");
   });
   ...
}

This code never sends an end signal and therefore will never respond. If you were to add a call to res.end() within that event handler, you will only get the first ping – which is the expected behavior because you are ending the response stream after the first chunk of data from stdout.

Code Sample 2:

spw.stdout.pipe(res);

Here stdout is flushing the packets to the browser, but the browser will not render the data chunks until all packets are received. Thus the reason why it waits 10 seconds and then renders the entirety of stdout. The major benefit to this method is not buffering the response in memory before sending — keeping your memory footprint lightweight.

edited Jun 20 '20 at 09:12

Community

1
1

answered Feb 21 '14 at 14:31

srquinn

10,134
2
48
54

thanks for your answer.. actually I have a a `res.end` in `spw.on('close'...`, but just didn't write it down. I was thinking the same with you, but this answer confused me: http://stackoverflow.com/questions/20357216/stream-stdout-from-child-process-to-browser-via-expressjs – anvarik Feb 21 '14 at 14:37
That answer is about streaming the data to the server in chunks to avoid buffering it all in the server first. Whether or not it is *rendered* in chunks is entirely up the the browser logic. – loganfsmyth Feb 21 '14 at 15:47
Right — the server sends it all out on the pipe, but the browser will not close the connection until all packets are received. I changed my explanation to reflect this – don't know what I was thinking before! – srquinn Feb 21 '14 at 15:58

nodejs/express - stream stdout instantly to the client

3 Answers3

Code Example 1:

Code Sample 2:

Linked