Compile .proto files on every directory

Question

I have a directory structure like this:

Makefile
README.md
go.mod
google/
  -> Protobuf dependencies (ignore)
gopb/
  -> Compiled Go protobuf files (ignore)
protoc-gen-openapiv2/
  -> Protobuf OpenAPI depdencies (ignore)
providers/
  - polygon/
    - tickers.proto
    - query.proto

I'd like to setup a single command that can find all directories with .proto files (except the ones indicated above) and run protoc on each of these, with output going to a directory of the same name under the gopb directory. So far, I have been able to create a find command that returns the directory names:

$ find -not \( -path "*.git*" -or -path "*google*" -or -path "*protoc*" -or -path . -or -path "*gopb*" \) -type d -print

./providers
./providers/polygon

Obviously, this is not entirely accurate and I'd like to fix that, but the main issue is that, when I use the -exec flag to call protoc:

$ find -not \( -path "*.git*" -or -path "*google*" -or -path "*protoc*" -or -path . -or -path "*gopb*" \) -type d \
> -exec protoc --go_out=./gopb/ --go_opt=paths=source_relative {}/*.proto \;

This prints two errors:

Could not make proto path relative: ./providers/*.proto: No such file or directory
Could not make proto path relative: ./providers/polygon/*.proto: No such file or directory

I'm not sure what I'm doing wrong here because, if I call protoc --go_out=./gopb/ --go_opt=paths=source_relative ./providers/polygon/*.proto, it creates the files as expected. What am I doing wrong here?

So you want to run `protoc` with all the `.proto` files of each directory? — Fravadona, May 25 '22 at 08:47
_"with output going to a directory of the same name under the gopb directory"_ What does that mean exactly? `gopd/providers/polygon`? — Fravadona, May 25 '22 at 09:52

score 1 · Answer 1 · answered May 25 '22 at 12:19

1

There are various ways to tackle the problem, but for *.proto to be treated as a glob you need to interpret it in a shell.

Here's how you could do it with find and bash:

find . -not \( ...paths... \) \
       -type d \
       -exec bash -c '
           shopt -s nullglob
           for d
           do
               f=( "$d"/*.proto )
               (( ${#f[@]} > 0 )) || continue
               protoc --go_out=./gopb/ --go_opt=paths=source_relative "${f[@]}"
           done
' _ {} +

answered May 25 '22 at 12:19

Fravadona

13,917
1
23
35

Sorry, but what do you mean, "treated as a glob?" I don't understand that bit – Woody1193 May 25 '22 at 22:46
1

Well, I mean that `find` won't expand the glob in `{}/*.proto`, it will only replace `{}` with the target; that's why you get the error `./providers/polygon/*.proto: No such file or directory` (you would get exactly the same result with `protoc ... "./providers/polygon/*.proto"`) – Fravadona May 25 '22 at 23:30
Is there a way I can do this without calling into a shell? I'm planning to embed this into a Makefile – Woody1193 May 26 '22 at 00:46
I decided to go a different route on this one but your explanation of globs gave me the hint I needed for that so I upvoted your answer. Thank you for your help. – Woody1193 May 26 '22 at 01:22

score 0 · Accepted Answer · answered May 26 '22 at 01:19

With help from @Fravadona, I realized that the method I was using to compile my .proto files wasn't going to work. So, I moved all my protobuf files that I wanted to compile into a new directory called protos, which modified my directory structure to:

Makefile
README.md
go.mod
google/
  -> Protobuf dependencies (ignore)
gopb/
  -> Compiled Go protobuf files (ignore)
protoc-gen-openapiv2/
  -> Protobuf OpenAPI depdencies (ignore)
protos/
  - providers/
    - polygon/
      - tickers.proto
      - query.proto

and then I modified the command to be this:

find ./protos -type f -name "*.proto" -exec protoc --go_out=../../../ {} \;

This new command finds all .proto files in the /protos directory and calls protoc on each. I removed the source-relative option which also required some code changes in my imports as well. But, this works for my needs. Note I had to use ../../../ for my output directory because my protobuf repo exists inside of github.com/myrepo/protobuf and I want the files written to github.com/myrepo/protobuf/gopb.

Compile .proto files on every directory

2 Answers2