3

I am doing some heavy statistic in node, so I forked the computation off to a child process where the child process will handle all the mongo query and loop through data etc and finally sends the result back to mother process. It's working all fine with relatively small data set.

But when the data gets large enough, the communication seems to be choked or something went wrong during the send back. The return data is fairly large HighChart Object which contains over 200k data point, I am able to log the message right before the child process invoke process.send(data), and the cpu activity just drop to almost 0 on the child process, which the mother process cpu usage got cranked way high, and just stay there and not dropping down. At that point the process seems just hang ( well not hang, cuz it's still responding to other request ), but the process.on('message',function(){}); is not triggered even I just let it sit there and run.

Is there a limit on the size where the IPC can handle for node? Is there anyway around this or some hack to it? Some people suggested to use TCP or other IP protocol instead of IPC, but it kinda complicates the issue with managing the child life span a bit.

Also is there any free tool that makes debugging node a bit easier beside just using console.log ?

Thank you very much for the help in advance

Regards. G

Gäng Tian
  • 1,588
  • 2
  • 18
  • 26
  • That sucks. I can't offer any help other than previous suggestions but would be interested in seeing if you find the issue/solutions. – Evan Shortiss Nov 06 '14 at 22:26
  • 1
    played around with different method of passing the data a bit, the one with least code change required is to use filestream to write teh stringyfy json to a file and once that's done pass the path to the file back to the master process and have it read it there then clean up.... it's slow, but at least things are moving now :/ – Gäng Tian Nov 07 '14 at 06:09
  • Yeah it's a workable solution but a bummer that IPC is an issue. – Evan Shortiss Nov 07 '14 at 14:21
  • @GängTian doing JSON.parse on the master is blocking operation, how do you get around that ? – Michael Oct 08 '15 at 09:35
  • 1
    actually I didn't parse json on the master thread, master thread simple feed the file stream to the front end and let browser handles the things there. It's blocking the browser, but at least it's the price user has to pay if they decided to over lay 200k dots without a proper aggregation. – Gäng Tian Oct 08 '15 at 19:36

2 Answers2

7

Folks, a few people are landing on this page and thinking that it is useful.

process.send() does not use System V IPC, so the information above by Penryn linking to System V message sizes is completely irrelevant.

There is no fundamental limit to process.send() message sizes, other than memory and cpu availability to create and json encode/decode such a message. As usual with node/javascript, JSON.parse() is synchronous, so calling it on extremely large buffers of JSON data will block node. If this is a problem, stream the JSON across a pipe, and use one of the streaming JSON decoding libraries - the overall performance will be poorer, but the overall responsiveness will be better because you won't get long blocking parses.

If anybody can find a reproduceable example of a so-called "message size limit", please report it as a bug to us at github.com/nodejs/node

All instances I've seen of this have been errors in the user's application code.

Sam
  • 87
  • 1
  • 2
  • It's also possible that the OP has run into this as yet unsolved issue related to IPC in node: https://github.com/nodejs/node/issues/9706 – Nic Barker Mar 16 '18 at 23:39
3

EDIT: Seems I am wrong. Please see Sam's message here: https://stackoverflow.com/a/40178217/1224279

My original answer follows below.

When node.js sends an object over IPC, I believe it is serialized into JSON, sent, and then deserialized.

There are message size limits. According to this page the limit is 8KB for a single message and 16KB for a message queue by default on Linux. You can change this by editing the respective /proc/sys/kernel files.

I'm not sure how that translates to Node (maybe it takes care of some buffering for you), but I'd guess there's some serious overhead in serializing/sending/deserializing 200k data points.

You might have better luck sending them in smaller bits. I've found a competent system can parse about 20k-80k small Node.js IPC messages every second.

If you're sending large amounts of data, you'd have to do exactly what you ended up doing: save data elsewhere and send its location, or use a different transport that's better-suited for gobs of data (such as pipes).

EDIT: See also: How to transfer/stream big data from/to child processes in node.js without using the blocking stdio?

Community
  • 1
  • 1
Penryn
  • 140
  • 8