2

I have been looking at protobuf-net recently and protocol buffers and so far it seems quite impressive. What i am wondering about is the service side of things. If i have understood the google documentation right you can declare services in the .proto files. My questions are

  1. Is this service support implemented in protobuf-net or another dotnet protobuf lib?
  2. Is the service support fully cross language like protocol buffers them selves are?
  3. How hard is it to generate clients at the other end based on the declaration declared in the .proto file?

basically i am considering setting up so web or tcp based services using protobuf if possible to reduce overheads when processing high volumes and protobuf seems ideal as my clients are generally implement in a mix of languages such as python, java, c++ and dotnet. I really want something that would be easy for both them and myself to integrate into their application and since i plan to use protobuf internally it will hopefully be easy to integrate on my side.

[edit] Just some extra info code.google.com/apis/protocolbuffers/docs/proto.html#services That is the type of service i was talking about but i am not sure what langs really support it and what you really get from the declaration in the .proto file.

Seer
  • 495
  • 5
  • 20

1 Answers1

3

If you look at the Third Party Add Ons page you'll find lots of separate RPC implementations. However, I don't believe Google has released an RPC implementation, and I'm not sure there are any particularly "standardised" ones in the third party collection... you'd really need to pick an RPC mechanism you liked, and probably do a bit of work to implement it anywhere it doesn't already exist.

(Disclaimer: I work for Google and own the protobuf-csharp-port project. However, I'm not writing this "on behalf" of Google. It's just my personal opinion.)

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • can you explain what the "service" instead of "message" in protobuf info on http://code.google.com/apis/protocolbuffers/docs/proto.html#services That seems to suggest there is one for some languages at least. The question is i guess what languages, how cross language it is and if there is dotnet server/client implementations and what languages also implement compatible clients – Seer Jan 18 '12 at 21:53
  • The service declaration part is cross-platform, and hopefully most implementations will implement it - but you need a specific RPC mechanism to plug it into. From that page: "The protocol compiler will then generate an abstract interface called SearchService and a corresponding "stub" implementation. The stub forwards all calls to an RpcChannel, which in turn is an abstract interface that you must define yourself in terms of your own RPC system." – Jon Skeet Jan 18 '12 at 21:58
  • oh now everything makes sense! Then each client would need to implement a RPC client that is compatabile with that individual server as since there is no standard for the RPC server there would be no standard for the client and therefore no way to generate the client cross language in the libraries been produced and the code generators. Is there a simple cross language RPC client/server implementation that would work with protobuf or is it likely i would need to build atleast part of it if i want it multi language? – Seer Jan 18 '12 at 22:03
  • @Seer: I don't know, to be honest. You should probably have a look at the projects listed on that page as a starting point. It really depends on what you need - authentication, repeatability, timeouts etc. It gets more complicated as you add more requirements :) – Jon Skeet Jan 18 '12 at 22:18
  • valid points. I am just looking for basic support in that they can just talk to each other and then investigate limitations but i have enough to get started i think – Seer Jan 19 '12 at 00:34
  • @Seer to add context; in most of my TCP usage, I'm content to just throw protobufs around as packets without any additional RPC services, and have the server figure it out based on the message; I also use HTTP routing which can make use of a few things. But in these cases I'm not really using the main services definitions at all - just throwing data around. – Marc Gravell Jan 19 '12 at 08:18
  • @Marc: Is there some sort of ID on each protobug message type when they are written? Is there a way to send many different protobuf messages over the same stream and work out what type they are when they arrive just from standard protobuf data or would i need to add some sort of custom data before each protobuf so the receiving end would know what type to use to deserialize the protobuf? I can think of ways to do this easily via a http server using different url but if done over tcp it would be slightly less standard. – Seer Jan 19 '12 at 20:01
  • @seer two easy ways; since you would typically be using the "*WithLengthPrefix" methods for an ongoing TCP conversation, you can use a different field-number per message type; Serializer.NonGeneric.TryDeserialize(...) allows you to map those back to the appropriate types. Perhaps easier, though, is to use inheritance of a common known message base-type, and use ProtoInclude to handle derived types. I can dig out examples of either if you want. – Marc Gravell Jan 19 '12 at 20:09
  • (tries not to look wounded at "protobug") – Marc Gravell Jan 19 '12 at 20:10
  • @Seer if you are talking to python, the field-header trick is the better idea. – Marc Gravell Jan 19 '12 at 20:11
  • @ Marc I am currently only dealing with dotnet clients but 95% chance i will have python clients and maybe a few java clients in the future that is why i want it to be as standard as possible so it is easy to implement. WithLengthPrefix is this standard so most implementations should support it easily? I assume the length prefix has nothing to do with length but is instead a sort of ID you can choose to use. Can this be defined in a .proto file or is it something you must set manually when sending. I suspect it is manual when sending but wanted to check. – Seer Jan 20 '12 at 00:33
  • @MarcGravell I would love to the examples you talked about above. When you said the field-header trick is the better idea is that the use a different field-number per message type or inheritance of a common known message base-type? – Seer Jan 21 '12 at 02:50
  • @Seer the "with length prefix" does both... Basically, protobuf reads a stream to the end, which is a big problem on a socket. This is not an official pattern, but I have seen people get the the same conclusion from various languages, so it is not uncommon either. Will dig out example. – Marc Gravell Jan 21 '12 at 08:53