4

I installed distcc and Cmake for a small C++ project. I have two machines: so there is one client where compilation is done locally, and one server, where the compilation is done 'remotely'.

I have followed the instructions to install and configure distcc but I see no improvement when I compile my project, using these simple steps:

CXX="distcc g++" cmake my_project && make -j4

I have checked everything, on the client and the server machines, and saw no particular behavior. All return codes are 0 (OK) in the logs and there are no error messages. It's like everything is working but with no time gain. I also installed the distccmon-gui tool and checked that both machines are used during compilation time.

Finally, I did try with 4 machines and got the same results, that is, 0 improvement.

The only thing which looks really weird, are the communication times:

COMPILE_OK exit:0 sig:0 core:0 ret:0 time:151ms 
COMPILE_OK exit:0 sig:0 core:0 ret:0 time:156ms 
COMPILE_OK exit:0 sig:0 core:0 ret:0 time:182ms 
COMPILE_OK exit:0 sig:0 core:0 ret:0 time:201ms 
COMPILE_OK exit:0 sig:0 core:0 ret:0 time:163ms 
COMPILE_OK exit:0 sig:0 core:0 ret:0 time:202ms

Even on the localhost machine, the latency is about 200ms for each performed job.

In the end, there is a little time overhead when i use distcc, which I attribute to communications between the server and the client.

Has anyone already had such an issue with distcc and knows where I should look, or where should I investigate? I'm really stuck on this and believe distcc should bring me additional performance!

Any ideas ? please help :=)

Thanks

Carmellose
  • 4,815
  • 10
  • 38
  • 56

3 Answers3

3

How many cores do you have in total?

Keep in mind that running make -j4 will use 4 cores in total: If there are 4 cores on your local machine and 4 on your server, you must use make -j8 or you'll have no improvement over just compiling locally.

Max Beikirch
  • 2,053
  • 5
  • 25
  • 36
3

Don't forget to set how many jobs the client is allowed to send to any given server.

Server/x # where server is the ip and x is the # of jobs

If you don't do this is defaults to 4.

Roge
  • 3,294
  • 2
  • 20
  • 19
  • This is an underrated answer! I was having a lot of trouble with only seeing 4 threads out of available 20 on my machine, and didn't see good documentation for the ~/.distcc/hosts file. Thank you! – Daniel Jul 12 '21 at 18:37
0

One issue with distributing builds is how well constructed the make files are.

If your makefile isn't constructed in such a way that it is actually possible to run in parallel, then using distcc probably wont make much difference.

Without distributing the build to another machine, do you get a speed up with -j 4 over no -j option? if not, you probably wont see a speed up just from off-loading, unless you are off-loading to a really fast machine, since it is likely only offloading a single job at a time

mjs
  • 2,837
  • 4
  • 28
  • 48
  • hi, unfortunately yes I see a definite improvement when I use parallel Make (-j4 or more) without distcc. – Carmellose Apr 11 '14 at 15:45
  • What is DISTCC_POTENTIAL_HOSTS set to? – mjs Apr 11 '14 at 20:19
  • on the client (localhost machine), it is set with the ips of the client and the server. It's something like: `export DISTCC_POTENTIAL_HOSTS='1.2.3.4 5.6.7.8'`, where the first ip is the locahost ip and the second the server ip. I have also set the `--allow` option in the server machine. When I do a `tail -f` of the logs of the client and server, and can see that they both communicate .. but there is no time improvement in the end `:-(` – Carmellose Apr 14 '14 at 09:23
  • You might try using localhost instead of the IP. That should make it use a loopback adapter, rather than going (potentially) all the way out onto the wire and back. – mjs Apr 14 '14 at 09:27
  • It goes the same with 127.0.0.1 and/or localhost. Also, I'm using `DISTCC_HOSTS` instead of `DISTCC_POTENTIAL_HOSTS` but it doesn't seem to have any impact. What's strange is the latency on the local host, which is huge .. it's around 200ms in any job summary. Do you know what could cause this ? – Carmellose Apr 14 '14 at 10:52
  • Nope. Sorry, I am out of ideas. – mjs Apr 14 '14 at 11:27
  • Argh .. Thanks anyway – Carmellose Apr 14 '14 at 12:16