http-proxy (reverse proxy) server hangs up in a random time

Question

Problem: When I send a request after this log (memory), it shows "socket hang up (connResetException)" error. And the NodeJS server stops working.

[1:0x330e8a0]    22724 ms: Mark-sweep 16.4 (25.2) -> 10.6 (29.9) MB,
 3.3 / 0.0 ms  (+ 0.1 ms in 2 steps since start of marking, biggest step 0.1 
ms, walltime since start of marking 7 ms) (average mu
 = 0.999, current mu = 1.000) finalize incremental marking
 via task GC in old space requested

Architecture: I use http-proxy as a reverse proxy. It enables the SSO(Single-Sign-On) and sends its traffics to the application.

Error Log in Proxy Server(Only happens in the production - more traffics)

/node_modules/http-proxy/lib/http-proxy/index.js:120
    throw err;
    ^

Error: socket hang up
    at connResetException (internal/errors.js:561:14)
    at Socket.socketCloseListener (_http_client.js:380:25)
    at Socket.emit (events.js:214:15)
    at TCP.<anonymous> (net.js:658:12) {
  code: 'ECONNRESET'
}

Since socket hangs up happens with MANY cases, I researched and experimented with various cases. And I'm thinking the memory leak issue from my node can be the issue. node --trace_gc src/index.js commands print 2~3 allocation failures per request.

Weird logs

 [1:0x449d780]    20364 ms: Scavenge 11.9 (13.2) -> 11.3 (13.2) MB, 3.2 / 0.0 ms \
     (average mu = 0.961, current mu = 0.961) allocation failure

Source Code

var apiProxy = httpProxy.createProxyServer();
app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json())

app.use(passport.initialize({}));
app.use(passport.session({}));
app.use(session({
    secret: 'secret',
    resave: false,
    saveUninitialized: true,}
));

app.get('/source*',
    function(req, res, next) {
        req.query.RelayState = req.url;
        if(req.user) {
            apiProxy.web(req, res, {target: xx});

Problem: [http-proxy expressjs] hangs up in a random time. It usually works for two days and the server goes 502 Error. I need to manually restart the container to restore it.

/node_modules/http-proxy/lib/http-proxy/index.js:120
    throw err;
    ^

Error: socket hang up
    at connResetException (internal/errors.js:561:14)
    at Socket.socketCloseListener (_http_client.js:380:25)
    at Socket.emit (events.js:214:15)
    at TCP.<anonymous> (net.js:658:12) {
  code: 'ECONNRESET'
}

Any idea is welcome... please help

Do you by chance have any memory traces for your proxy? What happens with node sometimes is, that you are leaking memory and slowly you run out of it and node kills itself. I would first check to see if being oom is a problem, than we can go further — huhnmonster, Jan 27 '20 at 10:02
I guess you do not have any logging setup. Also, I hope I understood correctly: You are using Express as a http proxy, right? I am just going to redirect you to this article, it has helped me out in the past https://marmelab.com/blog/2018/04/03/how-to-track-and-fix-memory-leak-with-nodejs.html Anyways, you are trying to find a sawtooth-like pattern in gc-pauses and/or memory usage. If you find one, it will probably be a memory leak. You can accelerate the test by setting up a container and running something like Apache Bench or attillery against it. Hope it helps — huhnmonster, Jan 27 '20 at 22:30
You can get the gc information on stderr by adding `--trace_gc` to the `node` command line if connecting a debugger to the app is hard. — Matt, Jan 28 '20 at 04:44
I was able to re-produce the error. Thank you guys. I see this `[1:0x3bdc780] 104466 ms: Scavenge 22.6 (24.2) -> 22.1 (24.4) MB, 7.8 / 0.1 ms (average mu = 0.999, current mu = 1.000) allocation failure` error on EVERY request. And NodeJS server went down after sending 40+ requests. @huhnmonster @Matt — merry-go-round, Feb 04 '20 at 17:39

http-proxy (reverse proxy) server hangs up in a random time

0 Answers0