How to change/remove/ignore a http header just before parsing the response in Node?

Question

I'm currently working with Node to get data from an API (Discogs API) and I've got this error when requesting :

{ [Error: Parse Error] bytesParsed: 0, code: 'HPE_INVALID_CONSTANT' }

Using this link : http://api.discogs.com/releases/249504 (but I have the same error for every other requests when content-length != actual content length)

And this code :

var http  = require('http');

var options = {
  hostname: 'api.discogs.com',
  port: 80,
  path: '/releases/249504',
  method: 'GET'
};

var req = http.request(options, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers);

  res.on('data', function(d) {
    console.log(d);
  });
});
req.end();

req.on('error', function(e) {
  console.error(e);
});

I found that the Content-length value is always smaller than the real response byte length.

Content-length : 2142

Actual Byte Length : 7,734 Bytes (according to https://mothereff.in/byte-counter)

I've read that Node's Parser is really strict and that's why it fails parsing the response.

Finally, I'm asking you if there is a way to either ignore/modify/delete a Header just before Node parses the response, so the parser can do his work by ignoring the Content-Length ?

score 1 · Accepted Answer · answered Aug 26 '15 at 10:30

This isn't related to a (perceived, see below) invalid content-length, because it also doesn't work using cURL:

$ curl  http://api.discogs.com/releases/249504
curl: (52) Empty reply from server

Apparently the API server requires that a user agent header is set:

var options = {
  hostname : 'api.discogs.com',
  port     : 80,
  path     : '/releases/249504',
  method   : 'GET',
  headers  : { 'user-agent' : 'foo/1.0' }
};

As for the difference in content-length: 2142 is the size of a gzip-compressed response (content-encoding: gzip), 7734 is the size of an uncompressed response. Apparently, your byte-counter test requests uncompressed responses only but the client that you checked the header with is requesting a compressed response.

Thanks to your simple answer I now undestand why it worked in my browser (because it sends an user-agent). And also why the length was not the same (before/after compression) — Lauromine, Aug 26 '15 at 11:10

score 1 · Answer 2 · answered Mar 23 '21 at 18:01

Moving back to the topic of changing headers just before parsing the response in Node, you can do it easily by replacing of standard handler with your own function on socket: "data" event.

https://github.com/nodejs/http2/blob/master/lib/_http_client.js#L655

var req = http.request(options, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers); // without "Pragma: public" header

  res.on('data', function(d) {
    console.log(d);
  });
});

// catch socket object on conection initialization
req.on('socket', function (socket) {
    // https://nodejs.org/api/events.html
    var standardHandler = socket.listeners('data')[0];
    socket.off('data', standardHandler);

    socket.on('data', function(data) {
        var str = data.toString();
        console.log(str);
        // HTTP/1.x 200 OK
        // Date: Sat, 28 Nov 2009 04:36:25 GMT
        // Connection: close
        // Pragma: public
        // Expires: Sat, 28 Nov 2009 05:36:25 GMT
        // Cache-Control: max-age=3600, public
        // Content-Type: text/html; charset=UTF-8
        // Content-Length: 533424
        // Content-Encoding: gzip
        // Vary: Accept-Encoding, Cookie, User-Agent
        //  
        // <!DOCTYPE html PUBLIC "-//W3C//DTD

        // Do anything you need
        // removing 4 line
        var lines = str.split('\r\n');
        lines.splice(3, 1)
        str = lines.join('\r\n');

        // Now pass new data onto standard request handler
        standardHandler.call(socket, Buffer.from(str, "utf-8"));
    })
});

req.end();

How to change/remove/ignore a http header just before parsing the response in Node?

2 Answers2