Get the whole response body when the response is chunked?

Question

I'm making a HTTP request and listen for "data":

response.on("data", function (data) { ... })

The problem is that the response is chunked so the "data" is just a piece of the body sent back.

How do I get the whole body sent back?

Answer 1

request.on('response', function (response) {
  var body = '';
  response.on('data', function (chunk) {
    body += chunk;
  });
  response.on('end', function () {
    console.log('BODY: ' + body);
  });
});
request.end();

Answer 2

Over at https://groups.google.com/forum/?fromgroups=#!topic/nodejs/75gfvfg6xuc, Tane Piper provides a good solution very similar to scriptfromscratch's, but for the case of a JSON response:

  request.on('response',function(response){
     var data = [];
     response.on('data', function(chunk) {
       data.push(chunk);
     });
     response.on('end', function() {
       var result = JSON.parse(data.join(''))
       return result
     });
   });`

This addresses the issue that OP brought up in the comments section of scriptfromscratch's answer.

Answer 3

I never worked with the HTTP-Client library, but since it works just like the server API, try something like this:

var data = '';
response.on('data', function(chunk) {
  // append chunk to your data
  data += chunk;
});

response.on('end', function() {
  // work with your data var
});

See node.js docs for reference.

Answer 4

In order to support the full spectrum of possible HTTP applications, Node.js's HTTP API is very low-level. So data is received chunk by chunk not as whole.
There are two approaches you can take to this problem:

1) Collect data across multiple "data" events and append the results
together prior to printing the output. Use the "end" event to determine
when the stream is finished and you can write the output.

var http = require('http') ;
http.get('some/url' , function (resp) {
    var respContent = '' ;
    resp.on('data' , function (data) {
        respContent += data.toString() ;//data is a buffer instance
    }) ;
    resp.on('end' ,  function() {
        console.log(respContent) ;
    }) ;
}).on('error' , console.error) ;

2) Use a third-party package to abstract the difficulties involved in
collecting an entire stream of data. Two different packages provide a
useful API for solving this problem (there are likely more!): bl (Buffer
List) and concat-stream; take your pick!

var http = require('http') ;
var bl = require('bl') ;

http.get('some/url', function (response) {
    response.pipe(bl(function (err, data) {
        if (err) {
            return console.error(err)
        }
        data = data.toString() ;
        console.log(data) ;
    })) 
}).on('error' , console.error) ;

Answer 5

The reason it's messed up is because you need to call JSON.parse(data.toString()). Data is a buffer so you can't just parse it directly.

Answer 6

If you don't mind using the request library

var request = require('request');
request('http://www.google.com', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the google web page.
  }
})

Answer 7

If you are dealing with non-ASCII contents(Especially for Chinese/Japanese/Korean characters, no matter what encoding they are), you'd better not treat chunk data passed over response.on('data') event as string directly.

Concatenate them as byte buffers and decode them in response.on('end') only to get the correct result.

// Snippet in TypeScript syntax:
//
// Assuming that the server-side will accept the "test_string" you post, and 
// respond a string that concatenates the content of "test_string" for many 
// times so that it will triggers multiple times of the on("data") events.
//

const data2Post = '{"test_string": "swamps/沼泽/沼澤/沼地/늪"}';
const postOptions = {
    hostname: "localhost",
    port: 5000,
    path: "/testService",
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
        'Content-Length': Buffer.byteLength(data2Post)  // Do not use data2Post.length on CJK string, it will return improper value for 'Content-Length'
    },
    timeout: 5000
};

let body: string = '';
let body_chunks: Array<Buffer> = [];
let body_chunks_bytelength: number = 0; // Used to terminate connection of too large POST response if you need.

let postReq = http.request(postOptions, (res) => {
    console.log(`statusCode: ${res.statusCode}`);

    res.on('data', (chunk: Buffer) => {
        body_chunks.push(chunk);
        body_chunks_bytelength += chunk.byteLength;

        // Debug print. Please note that as the chunk may contain incomplete characters, the decoding may not be correct here. Only used to demonstrating the difference compare to the final result in the res.on("end") event.
        console.log("Partial body: " + chunk.toString("utf8"));

        // Terminate the connection in case the POST response is too large. (10*1024*1024 = 10MB)
        if (body_chunks_bytelength > 10*1024*1024) {
            postReq.connection.destroy();
            console.error("Too large POST response. Connection terminated.");
        }
    });

    res.on('end', () => {
        // Decoding the correctly concatenated response data
        let mergedBodyChunkBuffer:Buffer = Buffer.concat(body_chunks);
        body = mergedBodyChunkBuffer.toString("utf8");

        console.log("Body using chunk: " + body);
        console.log(`body_chunks_bytelength=${body_chunks_bytelength}`);
    });
});

Answer 8

How about HTTPS chunked response? I've been trying to read a response from an API that response over HTTPS with a header Transfer-Encoding: chunked. Each chunk is a Buffer but when I concat them all together and try converting to string with UTF-8 I get weird characters.