NodeJS consuming Transfer-Encoding chunked using https.request

Question

I'm making a GET request to an API that responds with Transfer-Encoding: chunked. It's an https request, I've attached on('data') listener to it which then adds the response to an array.

There are around 5 chunks incoming each of them being a Buffer depending on the response. When I concat the array and try converting it to a String I get the weirdly decoded reply which looks to be encoded. I think it might be due to the request being HTTPS but I'm not sure, below's my code for decoding it

response.on('data', (data) => {
                    // Data is sent in binary chunks, let's add them up to an array
                    binaryDataArray.push(data)
                }).on('end', () => {
                    const buffer = Buffer.concat(binaryDataArray);

                    try{
                        const formattedData = JSON.parse(buffer.toString('utf8'));
                        resolveRequest(formattedData);
                    }catch(error){
                        setTimeout(makeRequest, 5000);
                    }
                });

Any help would be appreciated.

score 0 · Answer 1 · answered Oct 14 '22 at 07:46

Check your res.headers['content-encoding'], the response may be encoded by gzip. So you need to unzip the response buffer like this:

response.on('data', (data) => {
  // Data is sent in binary chunks, let's add them up to an array
  binaryDataArray.push(data)
}).on('end', () => {
  const buffer = Buffer.concat(binaryDataArray);

  zlib.gunzip(Buffer.concat(respData), function (err, decoded) {
    if (err) throw err;

    const formattedData = decoded.toString()
    resolveRequest(formattedData)
  });
});

NodeJS consuming Transfer-Encoding chunked using https.request

1 Answers1