1

I need to exchange both protobuf-net objects and files between computers and am trying to figure out the best way to do that. Is there a way for A to inform B that the object that follows is a protobuf or a file? Alternately, when a file is transmitted, is there a way to know that the file has ended and the Byte[] that follows is a protobuf?

Using C# 4.0, Visual Studio 2010

Thanks, Manish

Manish
  • 1,726
  • 3
  • 23
  • 29

1 Answers1

1

This has nothing to do with protobuf or files, and everything to do with your comms protocol, specifically "framing". This means simply: how you demark sub-messages in a single stream. For example, if this is a raw socket you might choose to send (all of)

  • a brief message-type, maybe a byte: 01 for file, 02 for a protobuf message of a particular file
  • a length prefix (typically 4 bytes network-byte-order)
  • the payload, consisting of the previous number of bytes

Then rinse and repeat for each message.

You don't state what comms you are asking, so I can be more specific.

Btw, another approach would be to treat a file as a protobuf message with a byte[] member - mainly suitable for small files, though

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • Thanks very much. Putting a file in a protobuf would be awesome. In your experience, what would be the max file size for which a protobuf would work well? "what comms you are asking" - sorry didn't understand what that means. Also, what would be a safe way to rinse and repeat? – Manish May 20 '12 at 20:35
  • @Manish to store internally it will want to use a byte[] - so the return question is: what is the biggest byte[] you are happy to use? If you run it outside of protobuf-net you can of course just use streaming - much better for huge files. Re "comms" - what are we talking here as the source/destination? Is this a flat binary file? A socket? Something inside WCF? Middleware like MSMQ? Need to know a bit more context to advise fully. – Marc Gravell May 20 '12 at 22:38
  • Thanks Marc. Oh, I get it. If we put the file inside the protobuf, we will have to declare say Byte[1000000] for ALL protobufs if we want to handle 1MB files. Very inefficient if files are varying in size. Got it. We are just using sockets, want to avoid frameworks such as WCF to avoid bloat. Files are ordinary Word/ Pdf/ Excel files. Good reference for streaming as recommended would be great. Multitude of MSDN code samples confuse selection of preferred method. – Manish May 21 '12 at 02:00
  • @Manish no, incorrect. There is no fixed sizing in protobuf. The problem is simply: big arrays are a pain. When you have a huge file, it could be a problem. – Marc Gravell May 21 '12 at 06:02
  • Thanks. In the original reply, you mention, "rinse and repeat". By rinse, do you mean stream.flush()? Or is there a better procedure to rinse. – Manish May 21 '12 at 11:04