0

I have some DigitalOcean droplet with small amount of RAM, but there I should to host a lot of pm2 processes (to be more accurate, 9 - 8 in cluster mode (with provided -i 0) and 1 in fork mode). These processes are NodeJS Express servers that do server-side rendering for React.js projects.

So, as expected, sometimes pm2 is crashing because there is not enough RAM, and after it crashing it obviously does not restore all processes despite of there is startup daemon enabled.

I know, that startup daemon works only on system starts, so now every time when pm2 crashes, I should manually go to the droplet and reboot the system on it.

Is there any workarounds to automate this process? Or, maybe there exist some pm2 daemon that works when pm2 starts?

Appreciate any help!
Thanks in advance!

Update

Here is the last crash log (the part, that is related to crash manually):

<--- Last few GCs --->

[11194:0x3184710] 38854201 ms: Scavenge 24.7 (28.2) -> 23.8 (28.2) MB, 0.5 / 0.0 ms  allocation failure
[11194:0x3184710] 38888980 ms: Scavenge 24.8 (28.2) -> 23.9 (28.7) MB, 0.7 / 0.0 ms  allocation failure
[11194:0x3184710] 39006864 ms: Scavenge 24.8 (28.7) -> 23.9 (28.7) MB, 0.7 / 0.0 ms  allocation failure
[11194:0x3184710] 39131478 ms: Scavenge 24.8 (28.7) -> 23.9 (28.7) MB, 0.4 / 0.0 ms  allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xa90e93a5879 <JSObject>
    0: builtin exit frame: parse(this=0xa90e93890a9 <Object map = 0x188eb7f82ba1>,0x33a45597fe41 <String[338]: {"type":"axm:monitor","data":{"Loop delay":{"value":"0.81ms","ag$

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
1: node::Abort() [node /home/<private path>/frontend/dist-server/server.bundle.js]
2: 0x8c20ec [node /home/<private path>/frontend/dist-server/server.bundle.js]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node /home/<private path>/frontend/dist-server/server.bundle.js]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node /home/<private path>/frontend/dist-server/server.bundle.js]
5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node /home/<private path>/frontend/dist-server/server.bundle$
6: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/<private path>/fronte$
7: v8::internal::Factory::NewJSObject(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::PretenureFlag) [node /home/<private path>/frontend/dist-server$
8: v8::internal::JsonParser<true>::ParseJsonObject() [node /home/<private path>/frontend/dist-server/server.bundle.js]
9: v8::internal::JsonParser<true>::ParseJsonValue() [node /home/<private path>/frontend/dist-server/server.bundle.js]
10: v8::internal::JsonParser<true>::ParseJson() [node /home/<private path>/frontend/dist-server/server.bundle.js]
11: v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::internal::Isolate*) [node /home/<private path>/frontend/dist-server/server.bundle.js]
12: 0x1d7ee3e8697d
Community
  • 1
  • 1
Limbo
  • 2,123
  • 21
  • 41
  • 1
    Could you just use less process in cluster, like instead of 9, 4 only? Can you show us the pm2 logs (~/.pm2/pm2.log) witch the exception that crash pm2? – Unitech Dec 17 '18 at 13:23
  • @Unitech I have updated the question with logs, thank you. Some time ago (about 3 months) all processes were run in fork mode, but there was crashes too (maybe, less frequently, to be honest). – Limbo Dec 17 '18 at 13:34
  • have you tried supervisor? http://supervisord.org/ – danielarend Dec 17 '18 at 13:42
  • Which system are running on? About supervisor, it does not handle automatic Node.js clustering – Unitech Dec 17 '18 at 13:43
  • @Unitech Ubuntu Server 16.04 – Limbo Dec 17 '18 at 13:45
  • @danielarend No. Ubuntu has it's own supervisors, but I'm too far from be familiar with this technologies) – Limbo Dec 17 '18 at 13:46

1 Answers1

1

Connect PM2 to PM2+ so the agent linking PM2 to PM2+ will act as a second watchdog. If the main PM2 crash, the PM2+ agent will respawn PM2 with the previous process list

Unitech
  • 5,781
  • 5
  • 40
  • 47
  • Thank you, I'll try to check it and mark you answer after it will help! :) – Limbo Dec 17 '18 at 13:47
  • Sure let me know if it solve the issue, I will try to reproduce that bug on a server – Unitech Dec 17 '18 at 13:47
  • Well, I have linked pm2 to this dashboard, but did not found where to manually set the watchdog for main pm2 process. Is it already set automatically? – Limbo Dec 18 '18 at 10:16
  • This dashboard is very powerful. Hovewer, as I understood, there is no possibility to show more than 4 processes without paid subscription? :) – Limbo Dec 18 '18 at 10:18
  • 1
    The watchdog is activated by default once you've linked PM2 to PM2+. Yes that is right, we have to pay our team to make all of this work! – Unitech Dec 18 '18 at 10:55
  • 1
    Okay, thank you! I surely understand, that your great work should be paid. Thank you for this service! Will wait for crash on pm2 to check :) – Limbo Dec 18 '18 at 10:59
  • Well, this did not work. After pm2 crash, it did not restart with projects, the projects list is empty, and projects are not listed in PM2+ :( – Limbo Dec 21 '18 at 08:29