3

Background:

  • I have a 128-core server which I would like to use as a build server.
  • I have a bunch of client machines which work with a not-so-new and not-so-powerful PC. (Can't upgrade! Not in my control.)

What I did:

  • I followed the distcc documentation.
  • And installed a virtual machine on the server with exactly the same compiler version, the same distcc version -- basically the same distribution, as on the client-machines.
  • After configuring the clients and the servers, I can remotely build. I can verify this using the distccmon-text tool. I can see on the server, there are 8 threads started by the distcc daemon and that are awaiting for build-jobs to come. This was good as a first step. You can see the output below to be sure.

enter image description here

  • Second Step: Since the client machines are dual-core machines while the server offers 128-cores, and not all clients will be compiling at the same time, I wanted to offload as much of the build as possible to the build-server.

Problems:

  1. First Problem: distcc, no matter how I try to configure it, always tries to distribute the build-jobs equally on the client and the server. Even though my configuration file looks as shown below:
# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
#
# By default, just test that it works in loopback mode.
# 127.0.0.1
172.24.26.208/8,cpp,lzo
localhost/0

Which as per the distcc documentation should give higher priority to the build-server and lower-priority to the localhost since it comes later in the list. Also, it should give 8 jobs to the build server and 0 jobs to the localhost. But no, that doesn't happen. Upon typing make -j8 what it tries to do is start 4 threads on localhost and 4 on remote. Not good. This you can see from the image below.

enter image description here

  1. Second Problem: What you would also notice is that the pre-processing is being done on the localmachine. For this there is a tool that comes with distcc. It is called the "distcc-pump" or the pump mode and can be used like this.
time pump make CC="distcc gcc" CXX="distcc g++" -j8

Unfortunately, pump mode or not, the pre-processing happens to be happening all on the localhost, as you can see from the above image. Sad.

Note: At no point does the distcc program, with the configurations I listed here, throw any errors nor warnings, neither on the server nor on the clients.

Versions:

  • gcc 4.4.5
  • distcc 3.2rc1.2

(Before someone suggests - "upgrade software!", newer versions are most likely not possible for me. Anyways, this version of distcc offers the features that I need. Also, I can upgrade the server virtual-machine but then there would be compiler version mistmatch between clients and the server. The clients I cannot upgrade.)

Any suggestions, feedback on how to improve this setup/(fix the problems) are most welcome.

Duck Dodgers
  • 3,409
  • 8
  • 29
  • 43

2 Answers2

0

EDIT : these solutions do not work, I let the answer to avoid someone else to propose again them

Try by

bruno
  • 32,421
  • 7
  • 25
  • 37
  • Ok. Thanks for the hints, @bruno. Let me try after lunch. I get back to you. – Duck Dodgers Apr 09 '19 at 10:58
  • Removing the `localhost` from the file containing hosts did not work. It would still give build-jobs to the the localhost. The 2nd part (vis-a-vis 127.0.0.1 instead of localhost), I still have to try. – Duck Dodgers Apr 09 '19 at 14:01
  • @JoeyMallone else do not use _distcc_ but `ssh 172.24.26.208 "cd ...; gcc -j8 ..."` ;-) – bruno Apr 09 '19 at 14:12
  • `...` three dots! `o.O`. Ok. As soon as I can get my hands on that machine to test again, I will give this a go as well. Thank you for time again @bruno! Much appreciated! – Duck Dodgers Apr 09 '19 at 14:17
  • I could finally get my hands on the test machine, and I tried a number of things, replacing the `localhost/0` line with `127.0.0.1/0` --> this sends all build tasks to localhost, even though the build-server is above it in the hosts list. `127.0.0.1/1` --> this has same effect as `localhost`, meaning builds are divided half and half. Removing the `localhost/127.0.0.1` line completely also distributes tasks half and half. – Duck Dodgers Apr 10 '19 at 07:45
  • @JoeyMallone ok, sorry that do not solve your problem, visibly there is a bug in _distcc_ – bruno Apr 10 '19 at 07:59
  • yes, seems so. I will try with newer versions. Thank you for your time @bruno. – Duck Dodgers Apr 10 '19 at 10:30
0

distcc actually differentiates between remote and local CPUs. But contrary to your interpretation, in the hosts file the IP address 127.0.0.1 is considered as a remote CPU and a distccd server is expected to be running there. Any number of jobs you define in the hosts file is interpreted only for these server nodes.

According to the man page, "localhost" is interpreted specially. This is what seems to not work for you. An alternative syntax is --localslots=<int>. Have you tested this?

Additionally, distcc runs jobs on the local host (the one where you start the driver program). First, all linking is done there. Second, when you specify a certain parallelism with make -jN, all jobs exceeding the available number of remote jobs are run on your local host, too - in addition to the workload distribution part of distcc. The option --localslots limits these. The man page does not mention localhost explicitly here. And then there are those jobs, which fail on the server and are repeated locally.

For the given 128-core server I would use the number of cores in the hosts file and start only that number of compile jobs:

$ cat ~/.distcc/hosts
172.24.26.208/128,cpp,lzo
$ make -j 128
...

Then I would expect to not see any compile jobs on the local machine.

The man page has some more words regarding recommended job numbers. Search for the section(s) starting with distcc spreads the jobs across both local and remote CPUs.

Matthias Doerfel
  • 1
  • 1