Node.js / Express video streaming (HTTP 206 Partial Content)

Question

I have a binary document (mp4 video file) in a database (MarkLogic). I am using the database's Node.js API to stream the document in chunks. The setup looks like this:

html file

<video controls="controls" width="600">
  <source src="/video/myvideo.mp4" type="video/mp4">
</video>

In express I have then setup a route which handles the /video/:param route (in the database the video has the unique identifier which is the string '/video/myvideo.mp4')

node.js

// I'm only showing the relevant things in here

const serveVideo = (req, res) => {
  var stream = db.documents.read('/gopro/malta.mp4').stream('chunked');

  var chunks = [];
  var chunkBytes = 0;
  var start = 0;
  stream.on('data', (chunk) => {
    var headers;
    var range = req.headers.range;
    var total = 214335483; //total length of vid in bytes

    if (range) {
      var chunkSize = chunk.length;
      // (start === 0) ? start = 0 : start += chunkBytes;
      if (chunkBytes === 0) {
        start = 0
      } else {
        start = chunkBytes + 1
      }
      chunkBytes += chunkSize;

      headers = {
        'Content-Range': 'bytes ' + start + '-' + chunkBytes + '/' + total,
        'Accept-Ranges': 'bytes',
        'Content-Length': chunkSize,
        'Content-Type': 'video/mp4'
      };
      res.writeHead(206, headers);
      chunks.push(chunk);
    }
  });
  stream.on('end', () => {
    var allChunks = Buffer.concat(chunks);
    res.end(allChunks);
  });
});


router.route('/video/:uri').get(serveVideo);

Now of course the above fails with 'Error: Can't set headers after they are sent.' which is all fair and square. But I can't get my head around this - the .stream('chunked') call forces the database to retrieve the document in chunks and I do see those chunks just fine, however how can I return a 206 for the browser? I can't do it in the .on('data') as data is being streamed so the header would be sent multiple times. I guess which database I'm using is not really relevant - I would like to understand the concept, or at least see what I'm doing wrong.

Any help is appreciated. All the examples and other discussions that I have seen that stream video using Node.js are reading the video file from disk.

update

Making a change to the code now allows FF to play the video but not Chrome:

let stream = db.documents.read({uris:'/gopro/malta.mp4'}).stream('chunked');
stream.pipe(res);

There are no errors in Chrome's console. Here are the header details - note that there are two requests for the mp4 file:

1st

Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:30 GMT
Transfer-Encoding:chunked
X-Powered-By:Express

Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=0-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36

2nd

Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:31 GMT
Transfer-Encoding:chunked
X-Powered-By:Express

Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=28-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36

Have you proven that your code above works by reading from the disk? I suggest that as the first step of troubleshooting. Once you break the back on that the general code is sound, then add the DB steps back in. — David Ennis -CleverLlamas.com, May 20 '16 at 06:43
Good question! I d say you should not call for writeHead, but that you ll need to dig into express to write header directly. You can also take a look to this [code](https://github.com/feross/webtorrent/blob/c85dd3b83bdbd7fcbbb32c65a080b7011e4daede/lib/server.js) which i know can stream a video with 206 responses, it is used in peerflix. see also https://gist.github.com/paolorossi/1993068 — , May 20 '16 at 06:50
You must most likely don't need all the chunk stuff. Set headers manually: `res.status(206);`. Then simply pipe the response: `let stream = db.yourChunkStuff(); stream.pipe(res);`. On a phone here so can't test, so only putting it in a comment now. — Zlatko, May 21 '16 at 08:35
Also a side comment: you use const and arrows and then a lot of vars and not lets, as is common in Es2015 codebases, I wonder why? — Zlatko, May 21 '16 at 08:39
@Zlatko - thanks. So the following: `let stream = db.documents.read({uris:'/gopro/malta.mp4'}).stream('chunked'); stream.pipe(res);` works in FF but not Chrome/Safari. If I add `res.status(206)` as well it doesn't work in either of those browsers. — Tamas, May 21 '16 at 16:19
@Zlatko - regarding the var/let - I was copying some old code and writing some new - I'll fix that later. :) — Tamas, May 21 '16 at 16:20
@tamas Got any error msgs from chrome? Network tab req and res headers details? — Zlatko, May 21 '16 at 17:00
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/112593/discussion-between-zlatko-and-tamas). — Zlatko, May 21 '16 at 23:45
@tamas Also added my comment as a real answer as it might come handy to people when they stumble on this question. — Zlatko, May 23 '16 at 09:05

score 4 · Answer 1 · answered May 20 '16 at 14:22

The on('data') callback could always have a closure over an isFirstChunk variable that is initialized to true and have a test that, if isFirstChunk is true, emits the headers and sets isFirstChunk to false.

It would be preferable to pipe the stream if possible. npm might offer a stream library (maybe even through2()?) that has an event when the first data arrives.

For the longer term, you could reasonably file an RFE for an event at the start of a stream.

Hoping that helps,

Zlatko · Answer 2 · 2016-06-23T14:44:22.080

3

You most likely don't need all the chunk stuff. Set headers manually:

res.status(206);

Then simply pipe the response:

let stream = db.yourChunkStuff();
stream.pipe(res);

The simplest way to is pipe streams.

edited Jun 23 '16 at 14:44

answered May 23 '16 at 09:04

Zlatko

18,936
14
70
123

Node.js / Express video streaming (HTTP 206 Partial Content)

2 Answers2