NodeJS network transmission gets linear higher with length

Question

The goal is to deliver files with a nodejs http server.

The problem is the transmission from the server response to the browser.

I deliver a string as content as shown in program code.

VPS: debian 7 / SSD / 2 core / ~30ms ping

Related: node.js response.write(data) taking long time when size of data is small

Some string size benchmarking (content-download time)

-   1 kb:   0,18 ms
-  10 kb:   7,00 ms
-  20 kb:  47,50 ms
-  30 kb:  55,00 ms
-  40 kb:  58,10 ms
-  50 kb:  86,20 ms
-  60 kb:  93,10 ms
-  80 kb: 107,10 ms
- 100 kb: 120,00 ms (nginx needs 0.31ms)

Program code:

// var request => the http incoming request
request.socket.setNoDelay();
request.connection.setNoDelay();

// var response => the response from the http incoming request

response.on('close', function () {
  console.log("[ABORTED][%s] %s", request.connection.remoteAddress, request.url);
});
response.on('finish', function () {
  console.log("[SUCCESS][%s] %s", request.connection.remoteAddress, request.url);
});

// get the content of the file (takes ~0ms)
var fileContent = fileSystem.readFileSync(filePath, 'utf8');

// Configure response header
response.writeHead(200, {
  'content-length': ''+fileContent.length,
  'content-type': 'text/plain',
});

// Send content
response.end(fileContent, 'utf8', function(){
  console.log("Ending response took %sms", Date.now()-timeB2);
});

Have you tried without the statsync and content length setting. — ArcSine, Apr 23 '16 at 13:49
@ArcSine when i don't use content-length, the timings are near 2-3 seconds for waiting (normal 40ms) and for download same — SEUH, Apr 23 '16 at 14:06
What if you cached the stat call so you didn't have to compute it on every serve? — ArcSine, Apr 23 '16 at 14:26
@ArcSine sorry i wasn't clear. The statSync call needs less than 1ms, so if i don't set content length, it only gets worse — SEUH, Apr 23 '16 at 15:11
Have you tested your code against a known good implementation of serving files, e.g. express's static content serving? It may be that there is internal overhead that you are running up against. — ArcSine, Apr 23 '16 at 15:23
@ArcSine tested with express example code (and express's sendFile function) but with same results. As said, nginx transfers the file in under 1ms, and nodejs hangs at a point but i don't find the problem — SEUH, Apr 23 '16 at 15:26
As far as I know nginx is written in C and has lower overhead and higher performance than Node (and the interpreter) — ArcSine, Apr 23 '16 at 17:00
@ArcSine That's right but when i simply do a respond.end();, nodejs is faster than nginx. The problem is the pipe to the response object — SEUH, Apr 23 '16 at 17:01

score 0 · Answer 1 · answered Apr 24 '16 at 20:16

TL;DR:

Change

response.writeHead(200, {
  'Content-Type': 'text',
  'Content-Length': stat.size
});

to

response.writeHead(200, {
  'Content-Type': 'application/arraybuffer',
  'Content-Length': stat.size
});

Long version: The problem is that the content-length is wrong. In fact, the information of the header is wrong as createReadStream returns a buffer(array of numbers), not text.

If you want to send as text, you can simply use

response.end(readFileSync (filePath, 'utf8'));

and it will automatically construct the proper header. Otherwise, you will need to find 'the number of characters' as put it as content-length and pass encoding to createReadStream, i.e.:

var readStream = fileSystem.createReadStream(filePath, 'utf8');

I don't know an easy way of acquiring 'the number of characters' in a text document.

The alternative is to send it as application/arraybuffer with the content-length as the byteLength of the buffer. When you use stat.size, it is actually the file byte size, which is equivalent to byteLength if you are to read it as a Buffer stream.

Good point. Tested it, but the problem is still there. I updated my question to clarify the problem — SEUH, Apr 27 '16 at 12:15

score 0 · Accepted Answer · answered Apr 30 '16 at 15:35

So, there was no problem.

The content download times are according to my network speed.

What made the difference was the compression and cache. Nginx ist faster because it uses some type of compression and caching.

To get better results, i added SPDY and GZIP to my nodejs application and the results are nearly the same as nginx.

NodeJS network transmission gets linear higher with length

2 Answers2