Import multiple protobuf files with common message names in a single proto file

Question

I have several protobuf files that share some message names and for this reason I cannot have all the generated go files in one folder as I run into issues like "..re-declared in this block". I attached a minimal go project how to reproduce this issue. The structure of the project is like below:

protos/myMessages/Folder0/MyMessage0.proto

syntax = "proto2";

package myMessages.Folder0;

option go_package = "myMessages/Folder0";

message Message0 {
    optional uint32 payload = 1;
}

protos/myMessages/Folder1/MyMessage1.proto

syntax = "proto2";

package myMessages.Folder1;

option go_package = "myMessages/Folder1";

message Message0 {
    optional uint32 payload = 1;
}

protos/myMessages/helloworld.proto

syntax = "proto2";

package myMessages;

option go_package = "./;gen";

import "myMessages/Folder0/MyMessage0.proto";
import "myMessages/Folder1/MyMessage1.proto";

service Greeter {
  rpc SayHello1 (myMessages.Folder0.Message0) returns (myMessages.Folder0.Message0) {}
  rpc SayHello2 (myMessages.Folder1.Message0) returns (myMessages.Folder1.Message0) {}
}

main.go

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "net"

    "google.golang.org/grpc"
    
    pb0 "main/build/myMessages/Folder0"
    pb1 "main/build/myMessages/Folder1"
    pb "main/build"
)

var (
    port = flag.Int("port", 50051, "The server port")
)

// server is used to implement helloworld.GreeterServer.
type server struct {
    pb.UnimplementedGreeterServer
}

func (s *server) SayHello1(ctx context.Context, in *pb0.Message0) (*pb0.Message0, error) {
    return &pb0.Message0{}, nil
}

func (s *server) SayHello2(ctx context.Context, in *pb1.Message0) (*pb1.Message0, error) {
    return &pb1.Message0{}, nil
}

func main() {
    flag.Parse()
    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterGreeterServer(s, &server{})
    log.Printf("server listening at %v", lis.Addr())
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

Makefile

all: build protos

protos:
    protoc -Iprotos --go_out=build protos/myMessages/Folder0/MyMessage0.proto
    protoc -Iprotos --go_out=build protos/myMessages/Folder1/MyMessage1.proto
    protoc -Iprotos --go_out=build --go-grpc_out=build protos/helloworld.proto
    go build -o build/server main.go

build:
    mkdir -p build

After I run the make command, I get the error below:

mkdir -p build
protoc -Iprotos --go_out=build protos/myMessages/Folder0/MyMessage0.proto
protoc -Iprotos --go_out=build protos/myMessages/Folder1/MyMessage1.proto
protoc -Iprotos --go_out=build --go-grpc_out=build protos/helloworld.proto
go build -o build/server main.go
build/helloworld.pb.go:12:2: package myMessages/Folder0 is not in GOROOT (/home/andrei/env/gosdk/go1.17.2/src/myMessages/Folder0)
build/helloworld.pb.go:13:2: package myMessages/Folder1 is not in GOROOT (/home/andrei/env/gosdk/go1.17.2/src/myMessages/Folder1)
make: *** [Makefile:21: protos] Error 1

The generated build/helloworld.pb.go and build/helloworld_grpc.pb.go files have the following imports:

import (
    Folder0 "myMessages/Folder0"
    Folder1 "myMessages/Folder1"
)

But if I change these to the following then it works:

import (
    Folder0 "main/build/myMessages/Folder0"
    Folder1 "main/build/myMessages/Folder1"
)

What's the correct way to organise this code to avoid manually adding the main/build prefix to the imports in the generated files?

Can you share a bit more about your use case? It looks like there might be some copying errors the way you've renamed things (ex. `myMessages.Folder1.MyMessage1` isn't declared anywhere). I'd also suggest following more standardized package naming and file organization conventions. I find these rules helpful: https://docs.buf.build/best-practices/style-guide#files-and-packages — Spencer Connaughton, Sep 05 '22 at 19:42
@SpencerConnaughton I will add a few more details to this question to rephrase. Basically I have a single proto file for gRPC services and loads of proto files in subfolders for gRPC messages. Two or more messages may share the same name: e.g. MyMessage1 in folder 1 and MyMessage1 in folder 2. I'm taking care of this in the proto file of gRPC by adding the package name to differentiate between them. — Andrei, Sep 06 '22 at 07:45
@SpencerConnaughton minimal example that triggers the problem added. — Andrei, Sep 06 '22 at 22:13
If the problem is simply that you don't want the `main/build` prefix, could you simply use `--go_out=.` instead of having the generated files go to the `build` folder? — Spencer Connaughton, Sep 07 '22 at 20:02

Spencer Connaughton · Answer 1 · 2022-09-08T17:35:17.810

I was able to get this to work by doing the following:

1. Get your module name from `go.mod`. In this instance, I'll assume it's `hello-world`

module hello-world
...

2. Configure the protos to build as subpackages of your module

protos/myMessages/Folder0/MyMessage0.proto

...
option go_package = "hello-world/myMessages/Folder0";
...

protos/myMessages/Folder1/MyMessage1.proto

...
option go_package = "hello-world/myMessages/Folder1";
...

protos/myMessages/helloworld.proto

...
option go_package = "hello-world/myMessages";
...

3. Generate the proto code in your project directory using `source_relative` paths

protoc -Iprotos --go_out=. --go_opt=paths=source_relative protos/myMessages/Folder0/MyMessage0.proto
protoc -Iprotos --go_out=. --go_opt=paths=source_relative protos/myMessages/Folder1/MyMessage1.proto
protoc -Iprotos --go_out=. --go_opt=paths=source_relative --go-grpc_opt=paths=source_relative --go-grpc_out=. protos/myMessages/helloworld.proto

You could also configure these and the go_package paths to use the build directory: use --go_out=build in this step, and change the go package paths in the previous and following steps to "hello-world/build/myMessages/<something>".

However, I think build is better for build products (not generated code), and it isn't how they do it in the official example code: https://developers.google.com/protocol-buffers/docs/gotutorial

4. Import the `myMessages` submodules as necessary

main.go

package main

import (
    ...

    pb "hello-world/myMessages"
    pb0 "hello-world/myMessages/Folder0"
    pb1 "hello-world/myMessages/Folder1"
)
...

I also suggest using proto3 instead of proto2 per: https://cloud.google.com/apis/design/proto3

To simplify developer experience and improve runtime efficiency, gRPC APIs should use Protocol Buffers version 3 (proto3) for API definition.

I also suggest using buf tools and/or building with bazel instead of calling protoc directly to avoid chasing these problems around in the future.

Import multiple protobuf files with common message names in a single proto file

1 Answers1

1. Get your module name from go.mod. In this instance, I'll assume it's hello-world

2. Configure the protos to build as subpackages of your module

3. Generate the proto code in your project directory using source_relative paths

4. Import the myMessages submodules as necessary

1. Get your module name from `go.mod`. In this instance, I'll assume it's `hello-world`

3. Generate the proto code in your project directory using `source_relative` paths

4. Import the `myMessages` submodules as necessary