protobuf: read a message in C++ from C#

Question

I am going to read messages that are stored consecutively in socket in C++ client that are sent from a C# server. I expect that I can read the size of a message like that:

google::protobuf::uint32 m;
coded_input->ReadVarint32(&m);
cout << m << endl;

Then I want to read the message:

Person person;
CodedInputStream::Limit limit = coded_input->PushLimit(m);
person.ParseFromCodedStream(coded_input);
coded_input->PopLimit(limit);

the C# code looks like:

var person = new Person { Id = 123456, Name = "Fred", Address = new Address { Line1 = "Flat 1", Line2 = "The Meadows ar garą " } };
Stream str = new NetworkStream(socket);
Serializer.SerializeWithLengthPrefix(str, person, PrefixStyle.Fixed32);

It doesn't work.

I get the C++ error (43 is result of cout << m << endl;) (id, name, address are all fields in Person)

43

libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of type "Person" because it is missing required fields: id, name, address

When I don't read the variant and parse the message straighforward from coded_input, all is fine (when I change SerializeWithLengthPrefix to serialize in server's code). However I need a method to distinguish consecutive messages, so I need to know the size of the message I'm going to read. I just don't know how to send the size.

I tried:

Stream buf = new MemoryStream();
Serializer.Serialize(buf, person);
ProtoWriter writer = new ProtoWriter(str, null);
ProtoWriter.WriteInt32((int)buf.Length, writer);

but then I get:

Unhandled Exception: ProtoBuf.ProtoException: Invalid serialization operation with wire-type None at position 0 at ProtoBuf.ProtoWriter.WriteInt32 (Int32 value, ProtoBuf.ProtoWriter writer) [0x00000] in :0
at protobuftest.MainClass.Main (System.String[] args) [0x00097] in /home/lorddidger/studia/csharp/protobuf-test/protobuf-test/Main.cs:31

I am not able to send any int that way. What is wrong?

---

Update: Actually, my aim is to find a way to transfer an integer (size) with protobuf. I need any example (both C++ and C#) how to handle that because I don't understand all details within protobuf.

I noticed that SerializeWithLengthPrefix send prefix (4 uint32) that look like: size 0 0 0. The size is number of bytes of the message after serialization (I guess). I thought PrefixStyle.Fixed32 says there is only one uint32 before the message but there are 4!

Finally, I thought you should use ProtoWriter.WriteInt32((int)buf.Length, writer) to transfer integers because I followed suggestion from somwhere in the internet which I suppose is a workaround relevant to C++. Now I see you shouldn't write varints - they are related to the engine and that is to sophisticated to spend time on.

Should I send a message with an int? How should I distinguish that from next message that size is stored in the first one?

I see C# can hadle prefixes fine but is there ANY COMPATIBLE with C++ and C# way to state size of a message? Otherwise, there is no **ing way to queue messages what I would find irrational.

Answer 1

Your first example includes just a length; the "with length prefix" actually encodes in a protobuf-compatible stream.

If you are decoding from c++, read two varints; the first is the field-number and wire-type; the second is the length. The first is packed as 3 bits wire-type, the rest is the field-number. You might also be able to specify field 0 to skip - I can't recall without checking.

---

Update; I checked this out, writing the data without a field-number (just a varint length prefix), and then reading it back in C# using two different APIs - individually with Deserialize, and as an enumerable block via DeserializeItems. I had to fix a bug in the latter, but the following will work from the next code push (note: only the reading code has a fix, so if you are writing in C# and reading in C++ this won't affect you):

using (var ms = new MemoryStream())
{
    // write data with a length-prefix but no field number
    Serializer.SerializeWithLengthPrefix(ms, new Foo { Bar = 1 }, PrefixStyle.Base128, 0);
    Serializer.SerializeWithLengthPrefix(ms, new Foo { Bar = 2 }, PrefixStyle.Base128, 0);
    Serializer.SerializeWithLengthPrefix(ms, new Foo { Bar = 3 }, PrefixStyle.Base128, 0);

    ms.Position = 0;
    Assert.AreEqual(9, ms.Length, "3 lengths, 3 headers, 3 values");

    // read the length prefix and use that to limit each call
    TypeModel model = RuntimeTypeModel.Default;
    int len, fieldNumber, bytesRead;
    List<Foo> foos = new List<Foo>();
    do
    {
        len = ProtoReader.ReadLengthPrefix(ms, false, PrefixStyle.Base128, out fieldNumber, out bytesRead);
        if (bytesRead <= 0) continue;

        foos.Add((Foo)model.Deserialize(ms, null, typeof(Foo), len));

        Assert.IsTrue(foos.Count <= 3, "too much data!");
    } while (bytesRead > 0);

    Assert.AreEqual(3, foos.Count);
    Assert.AreEqual(1, foos[0].Bar);
    Assert.AreEqual(2, foos[1].Bar);
    Assert.AreEqual(3, foos[2].Bar);

    // do it using DeserializeItems
    ms.Position = 0;

    foos.Clear();
    foreach (var obj in model.DeserializeItems<Foo>(ms, PrefixStyle.Base128, 0))
    {
        foos.Add(obj);
        Assert.IsTrue(foos.Count <= 3, "too much data!");
    }
    Assert.AreEqual(3, foos.Count);
    Assert.AreEqual(1, foos[0].Bar);
    Assert.AreEqual(2, foos[1].Bar);
    Assert.AreEqual(3, foos[2].Bar);
}