1

So I have a node (v8.9.4) server that is running on an AWS EC2 instance using the forever package to start it up. The server has worked without any issues for years but now that it's grown and more people are using it, it suddenly starts to time out all requests at seemingly random times, after working for a few hours.

I've found that running forever restart on the server gets all requests working again so I've got a temporary cronjob to restart it every hour but this is not good design and I would much rather have the server running without any issues.

I've gone through my server logs and found this which may be significant:

error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #131
Warning: connect.session() MemoryStore is not designed for a production environment, as it will leak memory, and will not scale past a single process.

Another thing that may be important, the server stays running while this issue occurs so any checks on the server status through UptimeRobot (or any server status checker) returns a success.

Considering the server will run fine for a few hours and also start up again with no issues after a restart, I'm thinking it is not an issue with the code but something else that I am not aware of. My current hypothesis is the requests will start timing out if the server runs out of CPU but I would like to explore more options before making the final call on the issue. If anyone had any insight into this issue, I would be super grateful! :)

s26haide
  • 49
  • 4
  • 2
    "*Considering the server will run fine for a few hours and also start up again with no issues after a restart, I'm thinking it is not an issue with the code but something else that I am not aware of*" - I disagree. It sounds like the code is, perhaps, leaking memory or file descriptors or some other resource. As it's never been stressed before you've just never seen it. Try to run it on a larger EC2 an see if it runs longer. – stdunbar Apr 20 '21 at 19:47
  • @stdunbar I see, that makes sense. This does seem to have classic symptoms of a memory leak. Any idea what the SIGKILL is coming from? Is that a common thing to see when too much memory has leaked? Thanks for the response! – s26haide Apr 20 '21 at 19:58
  • SIGKILL may indeed be sent when the system is very low on memory. However, the operating system doesn't know if it's a leak or if the process meant to allocate a great deal of memory. – stdunbar Apr 20 '21 at 20:19
  • Great, thank you so much! I really appreciate it :) I guess it's time to figure out how to deal with memory leaks hahaha this should be fun – s26haide Apr 20 '21 at 20:29

0 Answers0