Tons of issue with protoc

Question

I'm trying to make protoc output the generated files to a folder inside the folder where all my .proto files are. First of all, I hope we can agree that this commands are confusing and isn't very intuitive. I stumbled upon this documentation which seems to explain everything, but things doesn't work. So my protobuf files are inside a folder called proto. I have a few protobuf files that imports another protobuf file. I import them like this:

package proto;

option go_package = "github.com/me/golang-grpc-server/proto";

import "proto/another_proto.proto";

message Proto1 {
    ...
    ...
    repeated proto.AnotherProto another_proto = 6;
}

That should work right? Well, protoc doesn't know what to do with it. If I run this command.

protoc --proto_path=proto --go_out=out --go_opt=paths=source_relative --go-grpc_out=out --go-grpc_opt=paths=source_relative proto1.proto another_proto.proto

protoc would give me these errors:

proto/another_proto.proto: File not found.
proto1.proto:7:1: Import "proto/another_proto.proto" was not found or had errors.
proto1.proto:16:14: "proto.AnotherProto" is not defined.
proto1.proto:40:14: "AnotherProto" is not defined.

If I then put another_proto.proto in front of proto1.proto, I get these errors:

proto/another_proto.proto: File not found.
proto1.proto:7:1: Import "proto/another_proto.proto" was not found or had errors.
proto1.proto:16:14: "proto.AnotherProto" seems to be defined in "another_proto.proto", which is not imported by "proto1.proto".  To use it here, please add the necessary import.
proto1.proto:40:14: "proto.AnotherProto" seems to be defined in "another_proto.proto", which is not imported by "proto1.proto".  To use it here, please add the necessary import.

Then if I remove --proto_path=proto and add proto\ infront of every protobuf, I won't get those errors, but the generated files would be placed inside out\proto which is very confusing because I thought --go_opt=paths=source_relative tells protoc to place the outputs inside where the protobufs are placed, as stated here which in my case is the folder proto.

If the paths=source_relative flag is specified, the output file is placed in the same relative directory as the input file. For example, an input file protos/buzz.proto results in an output file at protos/buzz.pb.go.

I tried using --go_out=proto/out, but annoyingly it placed the generated files inside proto/out/proto. I'm guessing it's because of the proto/ infront of the protobufs, because of the previously mentioned errors, I might have to stick to out/proto for now as it's the currently most sensible path right now.

UPDATE:

I'm not quite sure if I understand @Daz correctly, but here's what my interpretation of your instructions.

So I made this changes to Proto1:

package src;

option go_package = "github.com/me/golang-grpc-server/proto/src";

import "src/another_proto.proto";

message Proto1 {
    ...
    ...
    repeated src.AnotherProto another_proto = 6;
}

I moved all the other protobufs inside a folder under proto and of course changed the go_package of each one to have /src at the end and changed the import parts as well to reflect the change. Then I modified my protoc command to this:

protoc --proto_path=proto/src --go_out=./out --go_opt=paths=source_relative --go-grpc_out=./out --go-grpc_opt=paths=source_relative proto/src/*.proto

That * was awesome. Same thing here, modified it to reflect the changes made to the folder structure. Unfortunately, the changes doesn't work, starting from the protobufs themselves. import doesn't seem to acknowledge the change in go_package or the package. If I use "src/another_proto.proto" as the import path, the linter would say that the path was "not found or has errors". Then when I use "proto/src/another_proto.proto", the errors would be gone, but when I run protoc, it would throw the previous "not found ..." errors.

UPDATE:

Man, this is some pretty infuriating situation which might be caused by a bug or just missing functionality on protoc go. So I was trying to push the command to place the stubs to a more logical I guess directory and so I modified the protoc command to have a longer --proto_path because I thought doing that could shave folders to the final output path. Each time I added a folder to the --proto_path, I modified the protobuf's go_package and import as well because errors about "not found" protobuf would appear. On my current command, the protobufs would look like this:

package src;

option go_package = "github.com/me/golang-grpc-server/proto/src";

import "another_proto.proto";

message Proto1 {
    ...
    ...
    repeated AnotherProto another_proto = 6;
}

which looks quite pretty. But, linter doesn't like it. I guess because of the fact, as @Daz mentioned, that protobuf doesn't really use Golangs package system, the linter doesn't know how to reference the import. Now, I have a project that I think should build, but has 2 files with errors. Here's my current command, this places the stubs to another folder in the same level as the proto folder:

protoc --proto_path=proto/src --go_out=out --go_opt=paths=source_relative --go-grpc_out=out --go-grpc_opt=paths=source_relative proto/src/*.proto

Answer 1

Yes, it's challenging.

The good news: once you understand the process, it will work consistently. I use the following approach with Golang, Rust and Python on Linux and it has worked well for me.

1. Protobuf sources, packages and folders

Protobuf source files define a package (!) hierachy.

I'll use Google's Well-known Types (WKT) by way of example. These files are in a protobuf (!) package google.protobuf. When you install protoc, generally these files are installed too in a folder called include and each WKT e.g. Timestamp.proto exists in a file-system folder derived from the package name (!):

{SOME-FOLDER}/include/google/protobuf/timestamp.pro

So Protobuf package paths map to file system folder paths but they must be anchored on some file-system folder (e.g. include in this case). This root folder {SOME-FOLDER}/include is the proto_path value in protoc. And package references from other proto sources that wish to import protos under this root folder are relative to this proto_path value as the root e.g. import "google/protobuf/timestamp.proto"; (no include but the folder path from this root must be reflected).

2. Compiled protos aka "stubs", packages and folders

Protobuf supports compilation to multiple languages and each has its unique quirks. Since you asked about Golang, I'll provide the Golang answer.

Protobuf and Golang share similarities in the package naming but it's important to remember that -- in many cases -- the protobuf source (!) package does not necessarily map to the language's packages.

For Golang, you must tell protoc the Golang packages that you want to use:

option go_package="example.com/path/to/package";

The WKT Timestamp is implemented by Google Timestamppb in google.golang.org/protobuf/types/known/timestampb package (entirely unrelated to google.protobuf

So...

In your example, I think you should remove on the proto references in your protobuf sources since this folder represents your project hierarchy not the protobuf package hierarchy.

Either as you have (!) the parent folder (!) of the file system folder proto becomes your proto_path because your package hierarchy starts at proto (or you have proto/proto which would be confusing):

package proto;

import "proto/another_proto.proto";

Or, as I recommend (!) you choose either a different package foo (or remove the package), continue to anchor proto_path on ${SOME_FOLDER}/proto and correct the import mechanism:

proto
└── foo
    ├── another_proto.proto
    └── some.proto

And some.proto:

package foo;

// Golang package is {GO-MODULE}/{GO-PACKAGE}
// Generally GO-PACKAGE={PROTO_PATH}/{PACKAGE} but it need not be
// You can change "${MODULE}/some/random/bar" here without problem
option go_package="github.com/me/golang-grpc-server/proto/foo";

// `another_proto.proto` is in the same `package` and thus same folder
import "foo/another_proto.proto";

// Proto1 is a message that ...
message Proto1 {
    ...
    ...
    // Type references must include the full `package` path e.g. `foo`
    repeated foo.AnotherProto another_proto = 6;
}

Using --go_opt=paths=source_relative is common but my preference (mostly because it makes sense to me) is the following:

# Golang Module name
# Protobuf sources will be `option go_package="${MODULE}/proto/foo";`
MODULE="github.com/me/golang-grpc-server"

go mod init ${MODULE}

# proto_path roots the source on "proto"
# Individual protos reflect the package in the folder under "proto"
# Using "foo" as the `package` name requires proto/foo/...
protoc \
--proto_path=${PWD}/proto
--go_out=${PWD} \
--go_opt=module=${MODULE} \
${PWD}/proto/foo/some.proto \
${PWD}/proto/foo/another_proto.proto

Because everything is under ${PWD}/proto/foo, you can just ${PWD}/proto/foo/*.proto too.

Yields:

.
├── go.mod
├── go.sum
└── proto
    └── foo
        ├── another_proto.pb.go
        ├── another_proto.proto
        ├── some.pb.go
        └── some.proto

Answer 2

Finally, thanks to @DazWilkin's initial answer, I finally found the command to do what I wanted to do. Here it is:

protoc --proto_path=proto/src --go_out=./proto/out --go_opt=paths=source_relative --go-grpc_out=./proto/out --go-grpc_opt=paths=source_relative proto/src/*.proto

I think the most important bit here is the --proto_path. Making it have the exact folder where my protobufs are, made me able to use whatever --go_out that I wanted. So protoc won't need to add the additional/leading folder that is missing in the --proto_path if it isn't the exact one.

Then on your protobufs, you'll have to do this:

package src;

option go_package = "github.com/me/golang-grpc-server/proto/src";

import "another_proto.proto";

message Proto1 {
    ...
    ...
    repeated AnotherProto another_proto = 6;
}

As mentioned on my last update, linter doesn't seem to like this because it doesn't know how to handle protobuf in this kind of setup probably.