What is the proper way to handle optional messages in protobuf?

Question

Consider the following implementation:

person.proto:

    enum PhoneType {
  MOBILE = 0;
  HOME = 1;
  WORK = 2;
}

message PhoneNumber {
  required float number = 1;
  required PhoneType type = 2;
}

message Person {
  required string name = 1;
  optional PhoneNumber number = 2;
}

main.cpp:

#include <stdint.h>
#include <stdio.h>

#include "person.pb.h"

#include <iostream>

int main(int argc, char *argv[])
{
    static const uint32_t outputbuflen = 1024;
    uint8_t outputbuffer[outputbuflen];
    uint32_t writtenlenght = 0;

    // Encode a person and send it over any datalink.

    // SENDER
    Person p_sender;
    p_sender.set_name("FooBar");

    // Set the phonenumber.
    // Does not work because returns const ref.
    // p.number().set_number(0123123);

    // Is this correct?
    PhoneNumber * number = p_sender.mutable_number();
    number->set_number(0800123123.0);
    number->set_type(MOBILE);

    p_sender.SerializeToArray(outputbuffer, outputbuflen);
    writtenlenght = p_sender.ByteSize();

    std::cout << writtenlenght << "B written to output."<< std::endl;

    // Great, now send the stream somwhere
    // sendStream(...);

    // RECEIVER
    // Does not know how many bytes to expect, so we'll let protobuf handle that.
    // Simulate a reception loop.
    Person p_receiver;
    uint32_t bytesparsed;
    for(bytesparsed = 0; bytesparsed < writtenlenght; bytesparsed++){
        // Look at the data as if receiving it byte by byte.
        std::cout << "Trying to parse message " << bytesparsed << std::endl;
        if(p_receiver.ParseFromArray(outputbuffer, bytesparsed)){
            std::cout << "Found message" << std::endl;;
            break;
        }
    }

    std::cout << "Needed " << bytesparsed << "B to read message, but has still " \
              << writtenlenght - bytesparsed << "B left in buffer" << std::endl;
    return 0;
}

Using: https://developers.google.com/protocol-buffers/ In a real world situation the receiver will not know the amount of bytes that should be received, but protobuf should (to my understanding) be able to check for a complete message from the buffer. This seems to work fine if there are no optional messages. At this moment it does not happen and I end up with a broken person with no phone number. :) What am I doing wrong and how should this situation be handled?

score 0 · Answer 1 · edited May 23 '17 at 12:20

0

Protocol Buffers serialization format does not cover framing (i.e. packing serialized messages to a stream or delimiting messages). You need to do it yourself.

However, there are some established practices. Check these two answers for details:

edited May 23 '17 at 12:20

Community

1
1

answered Jun 03 '12 at 18:41

alavrik

2,151
12
16

Take a look at my .proto -file. It has only one message I'm trying to send which has an optional message. It is not two different messages as in your links. – rotator Jun 03 '12 at 19:16
Actually I think i bumped into this: http://code.google.com/p/protobuf/issues/detail?id=377 – rotator Jun 03 '12 at 19:39
No, this is not the case. Regardless of whether you want to stream several messages or send only one, the reader _must_ know the size of the input message in order to correctly decode it. It'll become obvious if you read the encoding spec: https://developers.google.com/protocol-buffers/docs/encoding – alavrik Jun 03 '12 at 22:02
Shouldn't ParseFromArray return false in my example because the message is not complete? – rotator Jun 03 '12 at 22:57
ParseFromAray can not possibly know if a message is complete or not. You can satisfy it by giving it just required fields. The method you are chose for parsing messages is imprecise and very inefficient, see the links and explanations I posted earlier to learn how to do this properly. You should provide the exact size of the message you are about to parse. – alavrik Jun 04 '12 at 02:49
I still don't think you undestand the point. Lets say i write multiple messages to a buffer. I prefix these message with the length but one of these messages gets truncated somehow. Then I start reading the messages back by getting the size and the amount of bytes needed for one message and give it to the parser. The parser will say that the message is complete and the whole rest of the file will be corrupt. – rotator Jun 06 '12 at 18:08

What is the proper way to handle optional messages in protobuf?

1 Answers1