8

A limitation of the protobuf-net implementation is that it calls the underlying streams synchronously. By not offering an asynchronous API e.g. BeginSerialize/EndSerialize or a TPL equivalent, we are forced to tie up a thread waiting for synchronous stream I/O.

Is there any plan to offer asynchronous methods in protobuf-net, or alternatively, any creative ways around this problem?

Matt Howells
  • 40,310
  • 20
  • 83
  • 102

3 Answers3

8

No, that isn't currently supported and would be a lot of work.

My suggestion would be: buffer data yourself using the async APIs, and then when you have the data, use something like a MemoryStream to deserialize...

In my defence, I'm not aware of any other serializer that offers an async API here. In particular, when talking about slow/async streams, that usually means "network": and you usually have the issue of "framing" to consider there; protobuf-net won't know your framing requirements...

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • Is there any generic buffered stream that blocks on async? I'm currently using a MemoryStream approach but I want to partially deserialize the header of the file (as in http://stackoverflow.com/questions/13342318/deserialize-part-of-a-binary-file) and I have no way to know how many bytes I need to retrieve Async, only the deserializer knows. – tozevv Aug 16 '13 at 07:01
  • @tozevv sync-over-async? that *can* be fairly dangerous, to be honest. What is the underlying data source here? What async source is providing the data? A sync-over-async stream could probably be *written* easily enough – Marc Gravell Aug 16 '13 at 07:17
  • Yes, I have no problem with Writing. I'm reading from a file with a header and a body. If I know the size of the header is trivial to read the header (ReadAsync) first and only if the header meets a certain condition read the rest of the body also Async. Both the header and contents are being serialized using your excellent (thanks for that) implementation of protocol buffers. – tozevv Aug 16 '13 at 07:24
  • @tozevv actuall, the implementation is designed to not over-read without good reason, so you should be able to trust the max-length passed into `Read`; however, another option is to parse the start of the message manually (there are utilities for reading the various parts). Again, hard to be specific without context. – Marc Gravell Aug 16 '13 at 07:32
  • Added a detailed question http://stackoverflow.com/questions/18268442/partial-and-asynchronous-deserialization-c-sharp-with-protobuf-net – tozevv Aug 16 '13 at 07:52
  • I've submitted this as a feature request on protobuf-net issue tracker: https://code.google.com/p/protobuf-net/issues/detail?id=444 As can be seen from the proposal, it's not hard to do at all. – Robert Važan May 20 '14 at 20:10
3

I am using protobuff over the network. While the following solution doesn't guarantee it wont block, it does make life far better:

byte[] emptyByteArray = new Byte[0];
await stream.ReadAsync(emptyByteArray, 0, 0);
TaskData d = Serializer.DeserializeWithLengthPrefix<TaskData>(stream, PrefixStyle.Base128);

Because we make sure there is actual data on the stream before we start to deserialize, it will only end up blocking when the stream contains a partial message.

Edit: And we can use a similar trick for serialization:

MemoryStream mstm = new MemoryStream();
Serializer.SerializeWithLengthPrefix(mstm, data, PrefixStyle.Base128);
await stream.WriteAsync(mstm.GetBuffer(), 0, (int)mstm.Position);

As a bonus, this one does guarantee to never block.

Dorus
  • 7,276
  • 1
  • 30
  • 36
2

You can await Task.Run that will run the synchronous code in the thread pool. It's not the most efficient solution, but it's better than blocking. You can even submit your own CancellationToken to Task.Run:

await Task.Run(() => Serializer.SerializeWithLengthPrefix(
    stream, data, PrefixStyle.Base128), cancellationToken);

Alternatively, you can use a fairly simple helper method ripped from my JuiceStream library that I have submitted as part of async feature request to protobuf-net:

await ProtobufEx.SerializeWithLengthPrefixAsync(
    stream, data, PrefixStyle.Base128, cancellationToken);
await ProtobufEx.DeserializeWithLengthPrefixAsync<MyType>(
    stream, PrefixStyle.Base128, cancellationToken);
Robert Važan
  • 3,399
  • 2
  • 25
  • 31