0

I've set up a simple loop to poll an IronMQ messaging system, and everything works fine... except that memory usage increases more and more until it finally stabilizes at over 250MB. I've read that it's normal for Node to use more memory over time when run in a (sort of) recursive loop like this, even when running setTimeout and doing nothing, but I still don't understand the exact mechanics behind this behavior, or whether there is any way to control it. When making HTTP requests within the loop, memory usage more than doubles.

The code is running on a Heroku worker with a limit of 512MB RAM, leaving no breathing room to use cluster to use the rest of the available CPU cores. The memory usage can increase slowly or extremely quickly, depending on the jobs that run after receiving the messages.

This is the simplest code that reproduces this.

(function loop() {
  request.get('http://www.example.com', function(err, request, body) {
    if (err) console.log(err);

    setTimeout(loop, 200);
  });
})();

I've tried many, many ways of restructuring this code to prevent memory from increasing so high, but nothing has made any changes. Only the received HTTP response seems to have any effect on the upper limit of RAM used.

Is there a way to rewrite this entirely, or am I stuck with V8's behavior? All examples I've found use the same basic structure for infinite async loops, from kue to the async library.

ThreeHams
  • 125
  • 1
  • 7
  • 2
    Have you tried running the garbage collector manually at some points in your loop? – jfriend00 Nov 21 '14 at 01:30
  • Good call. I'd tried occasional collection before. This morning, I set it to collect every iteration, which keeps memory under control - at least, so far. This seems like a fairly bad idea for production code, though. Is there some way to structure this code so the request garbage is collected during scavenge instead of compaction? – ThreeHams Nov 21 '14 at 17:45
  • Presumably, GC is just not getting enough chances or enough cycles to run. I don't know why that is exactly, but you may learn something useful in this article: http://strongloop.com/strongblog/node-js-performance-garbage-collection/. Certainly it would be best to know why you're having GC issues, but absent that, I don't see anything wrong with reducing peak memory usage by calling for GC yourself at important times. You're just tweaking the optimization in favor of more frequent GC than the default design because that's what you want. – jfriend00 Nov 21 '14 at 18:06
  • Also, there are apparently ways to tweak the V8 GC settings in node: http://stackoverflow.com/questions/11472450/node-async-looping-why-does-the-memory-grow-and-just-drop-all-of-the-sudden and http://erikcorry.blogspot.dk/2012/11/memory-management-flags-in-v8.html – jfriend00 Nov 21 '14 at 18:06

0 Answers0