Receiving Protobuf3 Message over Network

Question

I'm utilizing Protobuf3 with c# for my client & c++ for my server, where the .proto's are generated from the corresponding protoc3 compiler. I'm new to utilizing Google Protocol Buffers, where I'm currently trying to figure out how to re-parse the received bytes on my server that were sent from my client to turn it back into it's originating google::protobuf::Message object on my c++ server.

My client is sending a CodedOutputStream in c# :

public PacketHandler()
{
  m_client  = GetComponent<ClientObject>();
  m_packet  = new byte[m_client.GetServerConnection().GetMaxPacketSize()];
  m_ostream = new BinaryWriter(new MemoryStream(m_packet));
}

public void DataToSend(IMessage message)
{
  CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream, true);
  output.WriteMessage(message);
  output.Flush();
  m_client.GetServerConnection().GetClient().Send(m_packet, message.CalculateSize());
}

This seems to be working, the message that is sent right now is a simple Ping message that looks like this:

// Ping.proto
syntax = "proto3";
package server;

import "BaseMessage.proto";

message Ping {
  base.BaseMessage base = 1;
}

message Pong {
  base.BaseMessage base = 1;
}

My BaseMessage looks like this :

// BaseMessage.proto
syntax = "proto3";

package base;

message BaseMessage {
  uint32 size = 1;
}

My plan is to hopefully extend all of my messages from a BaseMessage to allow for identification of the particular message eventually. Once I get the parsing & re-parsing figured out.

The received message that I am getting on my c++ server side looks like this : \u0004\n\u0002\b

When receiving the message I am attempting to re-parse using the CodedInputStream object by attempting to parse the received bytes.

PacketHandler::PacketHandler(QByteArray& packet, const Manager::ClientPtr client) :
    m_packet(packet),
    m_client(client)
{
  //unsigned char buffer[512] = { 0 };
  unsigned char data[packet.size()] = { 0 };
  memcpy(data, packet.data(), packet.size());

  google::protobuf::uint32 msgSize;
  google::protobuf::io::CodedInputStream inputStream(data, packet.size());
  //inputStream.ReadVarint32(&msgSize);
  //inputStream.ReadRaw(buffer, packet.size());
  server::Ping pingMsg;
  pingMsg.ParseFromCodedStream(&inputStream);
  qDebug() << pingMsg.base().size();
}

This is where I am a bit unsure of the process that is needing to be done to re-parse the message into the particular message. I believe if I utilize a BaseMessage that is extending all messages that will allow me to identify the particular message so I know which one to create. However, in this current test where I know it will be a Ping message, the ParseFromCodedStream doesn't seem to create the original message. My reasoning comes from during my qDebug() the pingMsg.base().size() is not the correct value that was set during the sending phase in my c# client.

Answer 1

I was able to get this solved by incrementing my sent byte count by + 1.

public void DataToSend(IMessage message)
{
  CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream, true);
  output.WriteMessage(message);
  output.Flush();
  (m_ostream.BaseStream as MemoryStream).SetLength(0); // reset stream for next packet(s)
  m_client.GetServerConnection().GetClient().Send(m_packet, message.CalculateSize() + 1);
}

I'm still a bit suspect as to why I need to add one. I'm thinking it must have to do with a null terminator. However prior without this addition my message would look like : #\u0012!\n\u001Ftype.googleapis.com/server.Pin

Then with the addition all it seems to do is add on is the correct message : #\u0012!\n\u001Ftype.googleapis.com/server.Ping

Also, during this process I figured out how to classify my messages utilizing the protobuf3 Any type. Where now my BaseMessage is defined as :

syntax = "proto3";

package base;

import "google/protobuf/any.proto";

message BaseMessage {
  uint32 size = 1;
  google.protobuf.Any msg = 2;
}

Which allows me to place any sort of google::protobuf::Message into the msg field, where on ParsingFromCodedStream I can check if it is that particular message.

PacketHandler::PacketHandler(QByteArray& packet, const Manager::ClientPtr client) :
    m_packet(packet),
    m_client(client)
  {
    unsigned char data[packet.size()] = { 0 };
    memcpy(data, packet.data(), packet.size());

    google::protobuf::io::CodedInputStream inputStream(data, packet.size());
    // read the prefixed length of the message & discard
    google::protobuf::uint32 msgSize;
    inputStream.ReadVarint32(&msgSize);
    // -----
    // collect the BaseMessage & execute functionality based on type_url
    base::BaseMessage msg;
    if (!msg.ParseFromCodedStream(&inputStream))
    {
      qDebug() << msg.DebugString().c_str();
      return;
    }

    if (msg.msg().Is<server::Ping>())
    {
      server::Ping pingMsg;
      msg.msg().UnpackTo(&pingMsg);
    }
  }