1

On the server side, I have this nodejs code. I simplified the code here to make the question clear:

var express = require('express');
var app = express();
var spawn = require('child_process').spawn;

app.get('/print_N', function(req, res) {
    var child = spawn('python', ['some_tick_data_process.py']);

    req.on('close', function() {
        console.log('req to close with pid=' + child.pid);
        child.kill('SIGTERM');
    });

    child.stdout.pipe(res);
    child.stdout.on('error', function(err) {
        console.log(child.pid + " error !!!")
        console.log(err);
    });
});
app.listen(3000);
console.log('Listening on port 3000...');

The underlying some_tick_data_process.py is quite I/O intensive. On the client side, I have a small python application to read the stream. The problem is that some processes will run into error "req to close". With a small number of processes, it is OK. I tried:

var http = require('http');
http.globalAgent.maxSockets = 100;

But it doesn't help. Please share your thoughts, thanks!

After leveraging on P.T.'s advice, the fix is:

app.get('/:version/query', function(req, res) {
    res.setTimeout(4 * 60 * 60 * 1000);  // nodejs socket timeout 4-hour
}
Liang
  • 21
  • 3
  • no you don't. There is no way you have python running in the browser, so that python is *also* running server side. Your node script is just spawning it. The whole point of node.js is that it can work async, and it's an infinitely running process, so you have no reason to escape to a python script, really. Just write your node code to do the right thing. (Node.js doesn't "stop running" once you write the response, or exit the `get` route handler. Any code you fire off async is still running.) – Mike 'Pomax' Kamermans Jun 05 '14 at 04:42
  • Mike, You are right that "some_tick_data_process.py" is running on the server side. I revised my post to make it clear. This process parses some data and each one usually finishes in one minute. – Liang Jun 05 '14 at 13:19
  • on an important note: don't make web requests spawn long-running processes. It's not what a web server is for, and makes your server a trivial DOS target. Please also explain what `some_tick_data_process.py` does, because it will almost certainly do something you should be using `cron` or the likes for, not overload a web request as trigger. – Mike 'Pomax' Kamermans Jun 05 '14 at 19:47
  • Hi Mike, we have a trading backtest system on the intranet, so the security is not a problem. The backtest code runs on researcher's desktop, but it needs the data from the server. The server uses node.js to pass the data to the backtest code. Any better proposal? – Liang Jun 05 '14 at 23:07
  • you might want to consider making the backtest runner its own python server, running on the researcher's desktop (so it's not limited by PHP's execution limits), and when a request comes in, you make your main server simply fire off a send-and-forget call to that python service instead. The researcher's machine runs the long-running task, without interfering with your real server. – Mike 'Pomax' Kamermans Jun 06 '14 at 02:54

1 Answers1

0

The express request 'close' events mean that the connection closed (I believe the express request inherits this from the underlying http.IncomingMessage.

This could be either because the client gave up, or because the nodejs socket timeout hit 2 minutes.

You need to figure out which side of the connection is giving up. I would've thought it was the client side, but was surprised to discover the nodejs timeout. See http://nodejs.org/api/http.html#http_request_setsocketkeepalive_enable_initialdelay for setting the socket timeout on the request's socket.

See also: node.js http.IncomingMessage does not fire 'close' event

Community
  • 1
  • 1
P.T.
  • 24,557
  • 7
  • 64
  • 95