memory error in node JS (node::smalloc::Alloc)

Question

I'm new to node Js, I've build a really simple server that send me back a zip file I request. It's all working but after some request a crash occur and i visualize this message on the terminal :

FATAL ERROR: node::smalloc::Alloc(v8::Handle, size_t, v8::ExternalArrayType) Out Of Memory

var http = require('http');
var url = require('url');
var fs = require('fs');

var port = 1337;


// create http server
var server = http.createServer(function (request, response) {

 var path = require('url').parse(request.url, true);
 console.log('requested ' + path.pathname);

 //get zipped resoures
 if (path.pathname == '/getzip') {
  console.log(request.url);
  var queryData = url.parse(request.url, true).query;
  if (queryData.name) {
   var filename = queryData.name;
   //open corrisponding file
   var zipFile = fs.readFileSync('packets/' + filename);
   response.writeHead(200, {
          'Content-Type': 'application/x-zip',
          'Content-disposition': 'attachment; filename=data.zip'
     });
     //send file in response
   response.end(zipFile);
  }
  else {
   response.writeHead(200, {'Content-Type': 'text/plain'});
    response.end('{error = "bad url"}');
  }
 }

}).listen(port);
server.timeout = 1000000;

Do you have any idea of what it can be? this code looks so simple.

How large is your zip file? Can you determine which line causes the allocation error? My suggestion is to place some output around the readFileSync and around response.end() to see if that's the line that's causing it. It might be really an out of memory issue if you have a lot of large files. — Octav Zlatior, Aug 06 '15 at 13:34
Also, don't always trust the garbage collector of node. It might be that your response object is not gc'd, so you might want to say response = null; at the end of your function to make sure it does not linger around. Shouldn't be the case, but you might want to try it anyway. — Octav Zlatior, Aug 06 '15 at 13:36
the biggest zip is a 500 mb; it's big but I'm running the script on 16GB ram machine. Thanks for the answer, now I make some test. — Alberto Scampini, Aug 06 '15 at 13:47
The crash happens in readFileSync (i check it using the log, as you said) I go to check the ram, if the problem is that the zip is too big you have some suggestion to send it as multiple packets in node ? — Alberto Scampini, Aug 06 '15 at 13:57
You could read it in chunks. You can get a file pointer with fs.open() and use fs.read on it (with position=null, to keep reading from where you left off). Check the return value of the function and read until it's 0, save the chunks to the response with .write() (I think, have a look at the API). But if it's a memory leak issue, it won't help. It will work if the file is just too large for readSync (maybe the system does not let you allocate 500+ megabytes of memory at once, I have never tried that :) ) — Octav Zlatior, Aug 06 '15 at 14:02
node fs API https://nodejs.org/api/fs.html#fs_fs_readsync_fd_buffer_offset_length_position — Octav Zlatior, Aug 06 '15 at 14:03
Thanks a lot; I will try it; also because the crash happens always when I request the 500 mb zip file (but not all the time, this is the strange thing :/) — Alberto Scampini, Aug 06 '15 at 14:41
Also, the answer below from robertklep looks pretty useful, I'd take that one. Since it only happens every now and again, I suspect some kind of memleak (or maybe just a bug in the v8 engine... or both :) ) How's the RAM status? You should not see any constant increase over time. — Octav Zlatior, Aug 06 '15 at 14:56

score 2 · Accepted Answer · answered Aug 06 '15 at 14:14

Instead of reading the entire file into memory, you should leverage streams for this:

response.writeHead(200, {
  'Content-Type'        : 'application/x-zip',
  'Content-disposition' : 'attachment; filename=data.zip'
});
fs.createReadStream('packets/' + filename).pipe(response);

memory error in node JS (node::smalloc::Alloc)

1 Answers1