0

A common strategy of sharing protocol buffers seems to be to store these files in a separate Git repository as recommended by multiple answers:

So I've decided to set up such a structure, but got stuck when setting up deployments.

My current situation

I have a service have called magic-service which implements a .proto file stored in proto-repo. So the magic-service repo does not store its own proto file, but fetches it when the service gets built.

Now I want to release a new feature for magic-service, but that requires the .proto file to be updated first. But of course the public protobuf file cannot be updated before the implementation is available. It's a circular dependency.

Question

How do I safely deploy my service without having to publish its proto file first?

I would think that this is a common problem, but I wasn't able to find anything about it. Any suggestions or resources are welcome!

Duncan Lukkenaer
  • 12,050
  • 13
  • 64
  • 97

2 Answers2

2

We have used similar structure in one of our Go language project. We have one proto repo on Git and other project are consuming those proto files from proto repository.

Instead of referring the latest proto file in Git, other projects are referring proto file with particular tag (version) in Git. So, we have never faced this circular dependency issue. May be this solution help you.

xs2tarunkukreja
  • 446
  • 3
  • 8
1

Updating proto files in distributed systems definitely requires a bit of finesse.

One strategy (mentioned in another comment) is to use releases. That way, you can have the server following a later release than the client.

Another option is to solve this through documentation. Proto files can, in fact, be released before the implementation has been deployed and is available. In a larger system, it’s a good idea to identify the field as not available yet in a comment.

Fields that have been set by the marshaller but are not known by the unmarshaller are treated as “unknown” fields. Updates must just be careful coordinated to ensure that that changes aren’t backward or forward incompatible.

  • Backward incompatible changes would mean that old versions of the type cannot be used to unmarshal messages generated by new versions of the type.
  • Forward incompatible changes would mean that new versions of the type cannot be used to unmarshal messages generated by old versions fo the type.

See the documentation for more information on updating proto types.

https://developers.google.com/protocol-buffers/docs/proto3#updating

As an example, a typical flow for adding a new field might follow this pattern:

  1. Add a new field to the proto file and document it as unavailable.
  2. Update the reading code to understand how to read the new field.
  3. Remove the documentation about it’s availability.
  4. Update the writing code to understand how to write the new field.
dolan
  • 1,716
  • 11
  • 22