1

I'm taking a glance over at the buildtools repo (https://github.com/bazelbuild/buildtools) and trying to understand the scope of its responsibilities as it relates to the three phases of a bazel build (loading, analysis, execution)

The repo's description states that it is A bazel BUILD file formatter and editor. I find much logic in the repo written in go-lang that lends complete support for an AST parser, starlark syntax interpreting capabilities, reformatting and rewriting of BUILD files and what not. Basically there's logic designed to operate upon a single starlark file at a time. Rereading that repo description in this light leads me to conclude that buildtools is really a single file scoped effort and presents tools that only intersect functionality wise (perhaps only partially) to those loading operations bazel conducts while building.

Question: Is it accurate that the focus of buildtools is upon the single starlark file?

If that's true then all the multiple starlark file analysis logic and so forth seems to actually be maintained over at https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/lib and I should not expect to find any tools for the analysis phase and beyond in the buildtools repo. Is that right?

jxramos
  • 7,356
  • 6
  • 57
  • 105

2 Answers2

3

I don't work on Buildtools, but we agree: these tools seem to focus on BUILD / .bzl files in isolation. They let you process these files in parallel, to do similar operations on them.

If you wonder whether these tools understand relations between these files, the answer seems to be no.

If you further wonder what tools do then, the answer is Bazel's query, cquery, and aquery. I'm not aware of a programmable API for these queries though; you have to run Bazel to perform them.

László
  • 3,973
  • 1
  • 13
  • 26
  • Thank you, some more corroborating evidence I stumbled across yesterday looking for something else. Jin's `awesome-bazel` page categorizes `buildtools` under the section [BUILD file tools](https://github.com/jin/awesome-bazel#build-file-tools), which immediately rings the _singular file_ bell in my mind. – jxramos Oct 29 '19 at 17:58
  • Come to think of it his section header is actually disambiguating too, focusing the notion of build onto the actual `BUILD` files themselves rather than the generic concept of _build_ as in tools for building software or dealing with build artifacts etc. Very good. – jxramos Oct 29 '19 at 18:17
1

buildtools has tools working on a syntactic level (it looks at the syntax tree). These tools are outside of Bazel and have no knowledge of Bazel build phases. In the future, we may expand the code to work on multiple files (for the static analysis), but it will still be independent from Bazel phases.

https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/lib/ is the source code of Bazel. The syntax/ directory includes the code for reading and evaluating the Starlark files. The code there is called by Skyframe. The interpreter is called by Skyframe many times in parallel, both during the loading and the analysis phases.

If you have a more specific question (what are you trying to do?), I can help more. :)

Laurent
  • 2,951
  • 16
  • 19
  • I was initially trying to query undisturbed Bazel definitions as defined in the original bazel file avoiding all the bazel transformations and macro replacements and renames downstream. I wound up writing a truncated go script off the `buildifier2` example that right after parsing the AST writes it to a json file and then I run that go script on all the startlark file in my repo. Then I load all those json files from python and do the data analysis in python. – jxramos Nov 05 '19 at 22:56
  • But then I discovered the rich xml output format which tacks a ton of information for querying off the xml and doing transformations from there https://docs.bazel.build/versions/master/query.html#output-xml. The label renaming is still present on some of the rules but the naming conventions adopted for custom rules makes the xml very intelligible. – jxramos Nov 05 '19 at 22:58
  • Actually just came across an example where a custom rule has a `attr.label_list` attribute where some values in that list don't get used when building a certain target and thus don't show up in the xml. I'm trying to query all these libraries built from that rule that have a certain value in their lists across the repo. I'm looking at one BUILD file that has that value but in my xml that value doesn't show up for that attribute because we don't build under that configuration. In general I want to query across multiple BUILD files from a static analysis sense sort of. – jxramos Nov 06 '19 at 01:07