Decode a prepended VarInt from a byte stream of unknown size in javascript/Nodejs

Question

I am writing a small utility library for me to request server status of a given minecraft host in js on node. I am using the Server List Ping Protocol as outlined here (https://wiki.vg/Server_List_Ping) and got it mostly working as expected, albeit having big trouble working with unsupported Data Types (VarInt) and had to scour the internet to find a way of converting js nums into VarInts in order to craft the necessary packet buffers:

function toVarIntBuffer(integer) {
    let buffer = Buffer.alloc(0);
    while (true) {
        let tmp = integer & 0b01111111;
        integer >>>= 7;
        if (integer != 0) {
            tmp |= 0b10000000;
        }
        buffer = Buffer.concat([buffer, Buffer.from([tmp])]);
        if (integer <= 0) break;
    }
    return buffer;
}

Right now I am able to request a server status by sending the handshake packet and then the query packet and do receive a JSON response with the length of the response prepended as a VarInt.

However the issue is here, where I simply don't know how to safely identify the VarInt from the beginning of the JSON response (as it can be anywhere up to 5 byte) and decode it back to a readable num so I can get the proper length of the response byte stream.

[...] as with all strings this is prefixed by its length as a VarInt

(from the protocol documentation)

My current super hacky workaround is to concatenate the chunks as String until the concatenated string contains the same count of '{'s and '}'s (meaning a full json object) and slice the json response at the first '{' before parsing it.

However I am very unhappy with this hacky, inefficient, unelegant and possibly unreliable way of solving the issue and would rather decode the VarInt in front of the JSON response in order to get a proper length to compare against.

Answer 1

I don't know this protocol, but VarInt in protobuf are coded with the MSB bit:

Each byte in a varint, except the last byte, has the most significant bit (msb) set – this indicates that there are further bytes to come. The lower 7 bits of each byte are used to store the two's complement representation of the number in groups of 7 bits, least significant group first.

Note: Too long for a comment, so posting as an answer.

Update: I browsed a bit through the URL you gave, and it is indeed the ProtoBuf VarInt. It is also described there with pseudo-code:

https://wiki.vg/Protocol#VarInt_and_VarLong

VarInt and VarLong Variable-length format such that smaller numbers use fewer bytes. These are very similar to Protocol Buffer Varints: the 7 least significant bits are used to encode the value and the most significant bit indicates whether there's another byte after it for the next part of the number. The least significant group is written first, followed by each of the more significant groups; thus, VarInts are effectively little endian (however, groups are 7 bits, not 8).

VarInts are never longer than 5 bytes, and VarLongs are never longer than 10 bytes.

Pseudocode to read and write VarInts and VarLongs:

Answer 2

Thanks to the reference material that @thst pointed me to, I was able to slap together a working way of reading VarInts in javascript.

function readVarInt(buffer) {
    let value = 0;
    let length = 0;
    let currentByte;

    while (true) {
        currentByte = buffer[length];
        value |= (currentByte & 0x7F) << (length * 7);
        length += 1;
        if (length > 5) {
            throw new Error('VarInt exceeds allowed bounds.');
        }
        if ((currentByte & 0x80) != 0x80) break;
    }
    return value;
}

buffer must be a byte stream starting with the VarInt, ideally using the std Buffer class.