1

I have some projects using Bazel, C++ and protobuf. I also use gitlab CI/CD to build, test, check coverage, etc.

The problem is that when the project compiles first time it also compiles a protobuf compiler, which adds about 15 minutes to each step (the step itself takes 1-5 min).

I was using a setup example from this documentation: https://blog.bazel.build/2017/02/27/protocol-buffers.html

Here I created a simple hello world example with protobuf. When I use protoc to generate *.pb.cc, *.pb.h files it takes about 5 seconds. When I use bazel build ... it takes 15 minutes, because it builds protobuf compiler.

Build log: https://gitlab.com/mvfwd/issue-bazel-protobuf-compile/-/jobs/1532045913

Main Question

Is there any other way to setup Bazel to use already precompiled protoc and to skip 15 min on every step?

Update 2021-08-27

Added overwriting proto_compiler and proto_toolchain_for_cc as described in Implicit Dependencies and Proto Toolchains

Building :person_proto now works fine

$ bazel build :person_proto
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
INFO: Analyzed target //:person_proto (19 packages loaded, 61 targets configured).
INFO: Found 1 target...
Target //:person_proto up-to-date:
  bazel-bin/person_proto-descriptor-set.proto.bin
INFO: Elapsed time: 0.428s, Critical Path: 0.08s
INFO: 5 processes: 4 internal, 1 linux-sandbox.
INFO: Build completed successfully, 5 total actions

but building :person_cc_proto fails

$ bazel build :person_cc_proto
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
ERROR: /home/m/Synology/drive/prog/2021/b/issue-bazel-protobuf-compile/BUILD:2:14: in :aspect_cc_proto_toolchain attribute of BazelCcProtoAspect aspect on proto_library rule //:person_proto: '@local_config_cc//:toolchain' does not have mandatory providers: ProtoLangToolchainProvider
ERROR: Analysis of target '//:person_cc_proto' failed; build aborted: Analysis of target '//:person_proto' failed
INFO: Elapsed time: 0.124s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 25 targets configured)
mvfwd
  • 141
  • 2
  • 9
  • From the list of [dependencies](https://github.com/bazelbuild/rules_proto/blob/master/proto/private/dependencies.bzl), it seems to me that the `protobuf_rule` uses a precompiled binary when it is available for the host architecture. – rds Aug 25 '21 at 17:26
  • In any case, you should set up a blaze remote cache, so that build artefacts are reused between CI invocations. – rds Aug 25 '21 at 17:26

1 Answers1

0

From https://blog.bazel.build/2017/02/27/protocol-buffers.html#implicit-dependencies-and-proto-toolchains

The proto_library rule implicitly depends on @com_google_protobuf//:protoc, which is the protocol buffer compiler. It must be a binary rule (in protobuf, it's a cc_binary). The rule can be overridden using the --proto_compiler command-line flag.

Hence, you could (not tested)

  1. add the precompiled binary in your workspace,
  2. define a cc_import for this target
  3. pass the --proto_compiler command-line flag
rds
  • 26,253
  • 19
  • 107
  • 134
  • I used github.com/google/startup-os as example, binded `proto_compiler` and was able to build `:person_proto`, but the next step still fails when I try to build `:person_cc_proto` with `every rule of type proto_library implicitly depends upon the target '@com_google_protobuf//:cc_toolchain'...` here are my changes in `dev` branch: https://gitlab.com/mvfwd/issue-bazel-protobuf-compile/-/tree/dev – mvfwd Aug 26 '21 at 19:53
  • I have overridden `protoc` and `:person_proto` is compiled successfully. Unfortunately building `:person_cc_proto` fails. Added an update to original post with more details. Could you please take a look? – mvfwd Aug 27 '21 at 07:50
  • From the error, the provided toolchain is not proto lang tool chain… Sounds like you wrote the toolchain yourself, and made a mistake there. – rds Sep 16 '21 at 10:43