How do I specify the number of CPU cores bazel uses?

Question

I am building syntaxnet (tensorflow fork) with bazel. It's working very slowly and keeps hanging up.

The last time I had this problem (with caffe) someone told me to change the number of cores I was using by adding -j4. In bazel this command did not work. Any custom command for bazel like this?

CPU Specs: 3.8ghz clock, Quadcore

CPU Model: AMD 4800(or something along those lines).

it should use all cores by default. IE, I'm running it on 12 core machine, and `top` shows 12 `clang` commands running in parallel and using all 100% CPU, that's not the case for you? — Yaroslav Bulatov, Jun 13 '16 at 19:00

score 3 · Accepted Answer · edited Jun 20 '19 at 14:33

3

I added --jobs 4 and it managed to get through. It's probably the same as -j4.

Complete command line was:

bazel test --jobs 4 --genrule_strategy=standalone syntaxnet/... util/utf8/...

edited Jun 20 '19 at 14:33

Vertexwahn

7,709
6
64
90

answered Jun 13 '16 at 19:54

dmansfield

1,108
10
22

aah okay. Much better than --local_resources!! – Rushat Rai Jun 14 '16 at 03:55
1

Note that, according to https://docs.bazel.build/versions/master/bazel-user-manual.html the default jobs value is 200. That acts as an upper bound for bazel, and it tries to work out what parallelism is possible based on memory, cpu count, etc. – Twirrim Sep 17 '17 at 17:11

score 0 · Answer 2 · answered Jul 20 '18 at 20:21

As Guy Coder describes in an answer to a similar question, try --local_resources. In my limited experience, it seems that Bazel consumes approximately double the number of cores I set and sometimes consumes all cores for many seconds. Even with bazel test -c opt --local_resources 20000,1,0.25 I see significant load on my system. (Following is his answer)

From Bazel User Manual

--local_resources availableRAM,availableCPU,availableIO

This option, which takes three comma-separated floating point arguments, specifies the amount of local resources that Bazel can take into consideration when scheduling build and test activities. Option expects amount of available RAM (in MB), number of CPU cores (with 1.0 representing single full core) and workstation I/O capability (with 1.0 representing average workstation). By default Bazel will estimate amount of RAM and number of CPU cores directly from system configuration and will assume 1.0 I/O resource.

If this option is used, Bazel will ignore both --ram_utilization_factor.

How do I specify the number of CPU cores bazel uses?

2 Answers2