1

I have a Makefile which has several targets. One of then it’s just to generate cpp code automatically. The second one is designated to compile those generated cpp.

I have configured a cluster of distributed compilation and I just want to specify a amount of jobs to use for the compilation of the cpp files themselves. And another different amount of jobs for the generation part.

Is it possible?

The idea is use “make -j30” to distribute by distcc the builds to several available hosts, but the target dedicated to generate code defined into the Makefile cannot be splitted in 30 jobs because this target just will be processed locally. And the localhost has not 30 cores to process them, so the machine freeze. That is the reason i need to define amount of jobs for the generation phase and another amount of jobs for the compilation phase.

ArAmIuS
  • 11
  • 3
  • 3
    I presume that you are referring to GNU make. You most likely have `gmake`'s complete documentation installed, and you can read what `gmake`'s `-j` option can or cannot do, straight from the horse's mouth. Is there any part of that documentation that's unclear to you? Have you already read the documentation, concluded that no such facility is possible using `gmake`'s built-in features, or you are unaware of existence of this documentation? – Sam Varshavchik Nov 25 '21 at 14:54
  • 1
    Depends, and it's hard to give you proper advice without seeing the relevant parts of your Makefile. If you know your Makefile will only ever be run with one active `make` job, you could code the number of jobs to spawn into the build rule. But if someone calls your Makefile with the `-j` option (and `make` *itself* runs with multiple jobs), that can become tricky. Why do you feel the need to use different job counts in the first place? – DevSolar Nov 25 '21 at 14:57
  • 1
    Why split it up? A compilation target is presumably depending on the generated source code and the generated source code target is depending on whatever it uses to generate the source code. Just describe all the dependencies and use `-j` as @SamVarshavchik suggests. I have `MAKEFLAGS += -j$(shell nproc) -Orecurse` at the top of one of my "main" Makefiles which then does `$(MAKE) -C $@` down into subdirectories described by a `$(SUBDIRS)` target. – Ted Lyngmo Nov 25 '21 at 14:58
  • I can’t show up here the content of the Makefile for confidential agreements. The reason to split up the jobs between different targets is due to the phase of generated code can’t be distributed by distcc but the compilation phase of the cpp code does. I read the documentation and I don’t see how to do that clearly. Perhaps I am missing something. – ArAmIuS Nov 25 '21 at 15:05
  • That you want to use `distcc` to distribute builds is an important detail that impacts what kind of answers you will address your needs. You should edit the question to describe how you are using `distcc` or how you want to use it. In the meantime, I have tagged it for you. – John Bollinger Nov 25 '21 at 16:32
  • Thanks. Is it better now? – ArAmIuS Nov 25 '21 at 16:53
  • No, it is not currently possible (and it would be surprisingly tricky to implement) ... at least *directly*. What you could do, however, is add bogus serialization between some of the generated files ... or even a simple "all compile jobs depend on all-generating-jobs, then have all-generating-jobs call a recursive `make` that uses a different `-j` option (might have to hide that, I'm not sure)" Also, are you sure it's "not enough CPU" that's freezing the generating process? More often it is "not enough RAM" or similar. I didn't consider is Guile scripting - I don't know what it can do. – o11c Nov 25 '21 at 17:05

1 Answers1

1

The idea is use “make -j30” to distribute by distcc the builds to several available hosts, but the target dedicated to generate code... will be processed locally. And the localhost has not 30 cores to process them, so the machine freeze

An alternative to manually setting a different static limit on the number of concurrent jobs is to let GNU Make throttle jobs dynamically by system load. See the -l or --max-load option documented here.

This should prevent local codegen jobs from overloading the system, while allowing the full number of distcc jobs to run in parallel (assuming they don't have much impact on the local system load - they shouldn't affect the run queue when they're blocked waiting on input from the remote build).

$ make -j30 -l5 ...

You'll have to choose a suitable load value for your system, and I can't help you with that. Try running top during a build and watch the load average section.

Useless
  • 64,155
  • 6
  • 88
  • 132