Cannot deserialize protobuf data from C++ in Java

Question

My problem is to serialize protobuf data in C++ and deserialize the data in Java probably. Here is the code I use to the hints given by dcn:

With this I create the protobuf data in C++ and write it to an ostream which is send via socket.

Name name;
name.set_name("platzhirsch");

boost::asio::streambuf b;
std::ostream os(&b);

ZeroCopyOutputStream *raw_output = new OstreamOutputStream(&os);
CodedOutputStream *coded_output = new CodedOutputStream(raw_output);

coded_output->WriteLittleEndian32(name.ByteSize());
name.SerializeToCodedStream(coded_output);
socket.send(b);

This is the Java side where I try to parse it:

NameProtos.Name name = NameProtos.Name.parseDelimitedFrom(socket.getInputStream());
System.out.println(name.newBuilder().build().toString());

However by this I get this Exception: com.google.protobuf.UninitializedMessageException: Message missing required fields: name

What am I missing?

The flawed code line is: name.newBuilder().build().toString()

This would have never worked, a new instance is created with uninitialized name field. Anyway the answer here solved the rest of my problem.

One last thing, which I was told in the protobuf mailinglist: In order to flush the CodedOutputStreams, the objects have to be deleted!

delete coded_output;
delete raw_output;

what does your code look like which fills the received array? — jtahlborn, Apr 15 '11 at 00:33
Try printing out the result of "message.SerializeAsString()" and of "received" and see if they're the same. — Kannan Goundan, Apr 15 '11 at 01:12
@jtahlborn I described it in my updated question. Please have a look. — Konrad Reiche, Apr 15 '11 at 09:04
@Kannan-Goundan They are. Please see also my modified question. — Konrad Reiche, Apr 15 '11 at 09:05
I do know nothing about protocol buffers, but are you sure that `SerializeAsString` and `toString` are the right methods here? Isn't there some binary serialization method? — Paŭlo Ebermann, Apr 15 '11 at 10:59
java and network byte order are big endian. seems a little odd that you are using little endian. — jtahlborn, Apr 15 '11 at 13:58

score 3 · Accepted Answer · edited May 23 '17 at 12:08

3

I don't know what received is in your Java code, but your problem may be due to some charset conversion. Note also that protobuf does not delimit the messages when serializing.

Therefore you should use raw data to transmit the messages (byte array or directly (de)serialize from/to streams). If you intent to send many message you should also send the size before you send the actual messages.

In Java you can do it directly via parseDelimitedFrom(InputStream) and writeDelimitedTo(OutputStream). You can do the same in C++ a litte more complex using CodedOutputStream like

codedOutput.WriteVarint32(protoMessage.ByteSize());
protoMessage.SerializeToCodedStream(&codedOutput);

See also this ealier thread.

edited May 23 '17 at 12:08

Community

1
1

answered Apr 15 '11 at 11:31

dcn

4,389
2
30
39

Thanks, this is helpful. Currently I serialize to std::ostream and send this via socket. name.SerializeToOStream(&os); and in Java I read it like this: NameProtos.Name.parseDelimitedFrom(socket.getInputStream()); but I guess it will not work without using CodeOutputStream? – Konrad Reiche Apr 15 '11 at 11:42
`parseDelimitedFrom` expects to read the size first, and I think `SerializeToOStream` does not write the size. If you only send a single message this is no problem (use `parseFrom` instead in this case). Otherwise you have to transmit the size too. – dcn Apr 15 '11 at 12:09
This is great, I have the feeling I am on the right way, however I still get exceptions on the Java side? Could you be so kind and look at the attachment I edited to my original question at the end? This is the code I am using currently, I don't know what I am missing. – Konrad Reiche Apr 15 '11 at 12:22

score 0 · Answer 2 · answered Apr 15 '11 at 14:45

0

You're writing two things to the stream, a size and the Name object, but only trying to read one.

As a general question: why do you feel the need to use CodedInputStream? To quote the docs:

Typically these classes will only be used internally by the protocol buffer library in order to encode and decode protocol buffers. Clients of the library only need to know about this class if they wish to write custom message parsing or serialization procedures

And to emphasize jtahlborn's comment: why little-endian? Java deals with big-endian values, so will have to convert on reading.

answered Apr 15 '11 at 14:45

Anon

2,328
12
7

This is what I am doing. I use protobuf for inter-process communication which means, I use it was messaging service. Further, I agree on the endian issue, but I cannot find API docs how to use alternative methods. – Konrad Reiche Apr 15 '11 at 15:37
@platzhirsch - not sure what that comment means. However, if you write two things to the stream, you need to read two things. And the code that you posted only reads one thing. – Anon Apr 15 '11 at 18:14

Cannot deserialize protobuf data from C++ in Java

2 Answers2

Linked