1

I'm using protobufs for serializing my data. So far I serialized my data on the server (node restify) send it, receive it (Request is made by XMLHttpRequest) and serialize it on the Client.

Now I want to employ zipping to reduce the transfered file size. I tried using the library pako, which uses zlib.

In a basic script that I used to compare the protobuf and zipping performance to json I used it this way, and there were no problems

var buffer = proto.encode(data,'MyType'); // Own library on top of protobufs
var output = pako.deflate(buffer);
var unpacked = pako.inflate(output);
var decoded = proto.decode(buffer,'MyType');

However if I try to do this in a client-server model I can't get it working.

Server:

server.get('/data', function (req, res) {
   const data = getMyData();
   const buffer = proto.encode(data, 'MyType');
   res.setHeader('content-type', 'application/octet-stream;');
   res.setHeader('Content-Encoding', 'gzip;');
   return res.send(200,buffer);
});

My own protolibrary serializes the data in protobuf and then deflates it:

...
let buffer = type.encode(message).finish();
buffer = pako.deflate(buffer);
return buffer;

The request looks like this:

public getData(){
    return new Promise((resolve,reject) => {
        const request = new XMLHttpRequest();
        request.open("GET", this.url, true);
        request.responseType = "arraybuffer";
        request.onload = function(evt) {
            const arr = new Uint8Array(request.response);
            const payload = proto.decode(request.response ,'MyType')
            resolve(payload);
        };
        request.send();
    });
}

The proto.decode method first inflates the buffer buffer = pako.inflate(buffer); and then deserializes it from Protobuf.

If the request is made i get following error: "Uncaught incorrect header check" returned by the inflate method of pako:

function inflate(input, options) {
  var inflator = new Inflate(options);

  inflator.push(input, true);

  // That will never happens, if you don't cheat with options :)
  if (inflator.err) { throw inflator.msg || msg[inflator.err]; }

  return inflator.result;
}

Also I looked at the request in Postman and found following: The deflated response looks like this: 120,156,60,221,119,64,21,237,119,39,240,247,246,242,10,49,191,244,178,73,54,157 and has a length of 378564

The same request without deflating (the protobuf) looks like this �:�: (� 0�8@H (� 0�8@H � (�0�8@�Hand has a length of 272613.

I'm assuming, that I'm doing something incorrectly on the server side, since the inflated request is larger than the one not using compression. Is it the content-type Header? I'm out of ideas.

Bernd Strehl
  • 2,852
  • 4
  • 24
  • 43

0 Answers0