Close stream when finished

Question

I am streaming HTML 5 video with Express using fs.createReadStream(path).pipe(res); which eventually causes the error Error: EMFILE: too many open files if I refresh the page too many times (in the thousands). I assume I need to perform some kind of cleanup and close the file to prevent this but am unsure what needs to be done.

I am creating a read stream every time the user seeks to a part of the video which the client has not loaded which makes this problem happen in a matter of minutes. Task manager shows the same file open hundreds of times so it appears to never get closed.

Relevant part of the stream function:

fs.stat(path, function(err, stats) {
  var size = stats.size;
  var range = req.headers.range;

  if (range) {
    var parts = range.replace(/bytes=/, '').split('-');
    var start = parseInt(parts[0]);
    var end = parts[1] ? parseInt(parts[1]) : size - 1;
    var chunksize = (end - start) + 1;

    res.writeHead(206, {
      'Content-Range': 'bytes ' + start + '-' + end + '/' + size,
      'Accept-Ranges': 'bytes',
      'Content-Length': chunksize,
      'Content-Type': 'video/mp4'
    });

    fs.createReadStream(path, {start: start, end: end}).pipe(res);
  } else {
    res.writeHead(200, {
      'Content-Length': size,
      'Accept-Ranges': 'bytes',
      'Content-Type': 'video/mp4'
    });

    fs.createReadStream(path).pipe(res);
  }
});

what os are you running on? Mac? Linux? etc... – Phil Poore Jan 07 '17 at 13:56 — Phil Poore, Jan 07 '17 at 13:56
@PhilPoore Windows, unsure if this happens on another OS. – user2248702 Jan 07 '17 at 14:16 — user2248702, Jan 07 '17 at 14:16

score 2 · Accepted Answer · answered Jan 08 '17 at 07:08

It turns out the stream only gets destroyed and the file closed when the end is reached - which rarely happens as you would have to load the entire file without seeking or closing the browser.

To fix this you have to listen for the close event of the response and destroy the stream:

res.on('close', function() {
  stream.destroy();
});

Sometimes the close handler is never called, even when it is placed on the first line in the request function. A fairly hacky way I found to fix this is to check if the socket is destroyed before creating the stream:

if (req.socket.destroyed) {
  return;
}

The fixed stream function:

fs.stat(path, function(err, stats) {
  if (req.socket.destroyed) {
    return;
  }

  var stream;
  var size = stats.size;
  var range = req.headers.range;

  if (range) {
    var parts = range.replace(/bytes=/, '').split('-');
    var start = parseInt(parts[0]);
    var end = parts[1] ? parseInt(parts[1]) : size - 1;
    var chunksize = (end - start) + 1;

    res.writeHead(206, {
      'Content-Range': 'bytes ' + start + '-' + end + '/' + size,
      'Accept-Ranges': 'bytes',
      'Content-Length': chunksize,
      'Content-Type': 'video/mp4'
    });

    stream = fs.createReadStream(path, {start: start, end: end});
  } else {
    res.writeHead(200, {
      'Content-Length': size,
      'Accept-Ranges': 'bytes',
      'Content-Type': 'video/mp4'
    });

    stream = fs.createReadStream(path);
  }

  stream.pipe(res);

  res.on('close', function() {
    stream.destroy();
  });
});

score 0 · Answer 2 · edited May 23 '17 at 12:09

0

On Linux/Unix EMFILE issues are resolved by ulimit. I'm not sure on windows. Google Windows ulimit EMFILE gets to this Stack Overflow

https://stackoverflow.com/a/729204/683017

Its saying 2048 file descriptors per process is limit on Windows. This would fit with what your saying

"in the thousands"

If this 2048 is a firm limit, and I'm not sure it is, (you'll have to google around to find out more about your specific OS).

Maybe think about rebalancing the load out to more processes, or more servers if possible.

edited May 23 '17 at 12:09

Community

1
1

answered Jan 07 '17 at 14:26

Phil Poore

2,086
19
30

The problem I'm experiencing is more of a cumulative effect. It is unlikely that I will ever be streaming more than a few files at once but many will be streamed during the run time of my application. I just need to work out how to close the files after I have used them. – user2248702 Jan 07 '17 at 16:35
On unix, and assume its similar with windows. File descriptors are cleaned up by OS as it needs to. File descriptors have a minimum timeout, to stop reopening the same file over and over... say each one stays open for a minimum of 30-60 seconds. So if you use 2048 in a minute, you'll run out and get Error you mention. This Error comes up a lot when benchmarking, but unusual to come up in everyday use. If it does, suggests something needs re-arranging to overcome the 2048 limit. – Phil Poore Jan 07 '17 at 16:38
I have taken a closer look at the files which are open and it appears that I can have the same file open hundreds of times by repeatedly streaming it (which will happen after a few minutes of seeking on a HTML 5 video). – user2248702 Jan 08 '17 at 02:56

Close stream when finished

2 Answers2