0

I'm trying to get protoc and protoc-gen-go to play nice in a multi-repo codebase using go modules.

I've managed to get things more or less working, until I introduced a major version bump in one of my "api" (i.e. protobuf) repos, and I've hit a bit of a barrier.

Here's a simplified picture of my setup:

Let's say I have two repos, github.com/kpruden/base-api and github.com/kpruden/dep-api.

base-api includes base-api.proto and base-api.pb.go generated from it.

Similarly, dep-api contains dep-api.proto and the corresponding generated code. In addition, dep-api.proto depends on base-api.proto via an import github.com/kpruden/base-api/base-api.proto statement.

Both base-api and dep-api use go modules, with dep-api's go.mod specifying a dependency on base-api.

To make this work, I can run go mod vendor in the dep-api repo to pull all of its dependencies into a vendor directory, then add -I./vendor to my call to protoc.

This all works well until I decide I need to do a major release of base-api. This is done by updating base-api/go.mod's module line to end with /v2. I have no need to keep version 1 around, so I don't create a v2 subdirectory in my repo, and the version 2 code and protobuf definitions exist at the root of the repo.

At first, this didn't seem to be a problem. Since I'm not moving any files, protoc can still find base-api.proto. However, in the generated go code, version 2 of base-api must be imported using import "github.com/kpruden/base-api/v2", but protoc is generating the code to import it at github.com/kpruden/base-api.

My question is: how do I get protoc to generate the go code for dep-api to import base-api at the correct module path? Is this even possible? None of the documentation of either protobuf or gRPC have much of anything to say about go modules. Google has led me to some github issues in these projects that talk about supporting go modules, but they are mostly internal discussions without much in the way of prescriptions, and I've found no mention anywhere of handling major version releases.

Kris Pruden
  • 3,280
  • 4
  • 25
  • 30
  • 1
    Does using the protoc mapping option as done here work for you: https://github.com/grpc/grpc-go/blob/0f73133e3aa3bf4505007ebd0cf3508dfc952a22/regenerate.sh#L80 Using this, you can control import paths in the generated code. – Easwar Swaminathan Aug 19 '20 at 17:34
  • This helps a lot, thanks! By chance do you know where the options to the `--go-out` generator are documented? – Kris Pruden Aug 19 '20 at 18:39

1 Answers1

0

I was able to get something working. The setup I describe in the question is actually pretty close to what ultimately ended up working.

In summary, things mostly work as expected:

  1. In the parent module, a major version bump is accomplished by appending /v2 to the module line in go.mod. Nothing else is required. In particular, the directory layout in source control doesn't need to have v2 anywhere.
  2. In the child module referencing v2 of the parent module in go code is handled normally. Using the example above, a simple import "github.com/kpruden/base-api/v2" will do it.
  3. When including the parent protobuf definitions, use the module path: import "github.com/kpruden/base-api/v2/base-api.proto"
  4. To be able to generate the bindings using protoc use go mod vendor to copy dependencies into ./vendor which will lay them such that the protobuf definitions can be found and the import path in (3). At this point protoc -I./vendor ... works as expected, the imports in the generated go source are correct, and everyone's happy :)
Kris Pruden
  • 3,280
  • 4
  • 25
  • 30
  • AFAIK including external dependencies in `./vendor` or `./third_party` is the right way to do this. The hardcore way would be to use something like big boy's tool - Bazel (I am trying to do this right now :) ) – Dzintars Sep 06 '20 at 22:08