Configuring distcc for makefiles

Question

I am testing distcc and I found this tutorial very easy to follow, I should say that I am able to compile llvm using CMake + Ninja and CMake + Makefiles using distcc and docker with no problems. Since I am particularly interested in Makefiles, I looked at the files produced by CMake + Ninja and all I saw was that distcc is added as a prefix to the compiler (e.g. distcc /usr/bin/gcc). So I thought that if I create a small project using only Makefiles I could possibly add distcc and things will magically work. Well, looks like I am missing something because all I get is:

$ ./mk.sh image

make[1]: Entering directory '/home/user/sandbox/distcc-makefiles'
Compiling Lib/Print.c
Compiling Boot/Main.c
distcc[16644] ERROR: compile Boot/Main.c on 172.17.0.2/1 failed
distcc[16644] (dcc_build_somewhere) Warning: remote compilation of 'Boot/Main.c' failed, retrying locally
distcc[16644] Warning: failed to distribute Boot/Main.c to 172.17.0.2/1, running locally instead
distcc[16644] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'Boot/Main.c' failed, retried locally and got a different result.
distcc[16643] ERROR: compile Lib/Print.c on 172.17.0.2/1 failed
distcc[16643] (dcc_build_somewhere) Warning: remote compilation of 'Lib/Print.c' failed, retrying locally
distcc[16643] Warning: failed to distribute Lib/Print.c to 172.17.0.2/1, running locally instead
distcc[16643] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'Lib/Print.c' failed, retried locally and got a different result.
make[1]: Leaving directory '/home/user/sandbox/distcc-makefiles'

By running the llvm example I discarded all docker connection and distcc configuration issues, I tried to search for any ideas/examples on the internet but most of the people use CMake which is not my case. Suggestions are welcome, I based my demo project on a real project I am working on, so that's how basically the Makefile look like.

I don't know what I am missing...

Versions:

Just for reference:

$ distcc --version
distcc 3.1 x86_64-pc-linux-gnu
  (protocols 1, 2 and 3) (default port 3632)
  built Oct 19 2017 11:05:18
Copyright (C) 2002, 2003, 2004 by Martin Pool.
Includes miniLZO (C) 1996-2002 by Markus Franz Xaver Johannes Oberhumer.
Portions Copyright (C) 2007-2008 Google.

$ gcc --version
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.

$ make --version
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.

Answer 1

After enabling debug messages (with $ export DISTCC_VERBOSE=1), I found that distcc actually compiles in the server but the compilation fails and that's why it retries locally.

It fails because I am using external files to keep additional options (see gcc @file) and distcc doesnt interpret that as a file but rather as a string basically. That's why the compilation fails and distcc fallsback.

Log inspection

For reference, the distcc log looks like the following (divided into sections for further explanation):

... {{ SECTION A }}
distcc[13039] (dcc_spawn_child) forking to execute: gcc -std=c99 -Wall @_out/Release/Boot/Main.o.via -E /home/user/sandbox/distcc-makefiles/Boot/Main.c
distcc[13040] (dcc_get_hostlist) read hosts from environment
distcc[13040] (dcc_parse_hosts) found tcp token "172.17.0.2/1"
distcc[13039] (dcc_spawn_child) child started as pid13041

... {{ SECTION B }}

distcc[13039] (dcc_strip_local_args) result: gcc -std=c99 -Wall @_out/Release/Boot/Main.o.via -c /home/user/sandbox/distcc-makefiles/Boot/Main.c -o _out/Release/Boot/Main.o
distcc[13039] exec on 172.17.0.2/1: gcc -std=c99 -Wall @_out/Release/Boot/Main.o.via -c /home/user/sandbox/distcc-makefiles/Boot/Main.c -o _out/Release/Boot/Main.o
distcc[13039] (dcc_note_state) note state 2, file "Main.c", host "172.17.0.2"
distcc[13039] (dcc_connect_by_name) connecting to 172.17.0.2 port 3632
distcc[13041] (dcc_new_pgrp) entered process group
distcc[13041] (dcc_increment_safeguard) setting safeguard: _DISTCC_SAFEGUARD=1
distcc[13039] (dcc_connect_by_addr) started connecting to 172.17.0.2:3632

... {{ SECTION C }}

distcc[13039] (dcc_select_for_write) select for write on fd6
distcc[13039] (dcc_note_state) note state 4, file "(NULL)", host "(NULL)"
distcc[13039] (dcc_x_token_int) send DIST00000001
distcc[13039] (dcc_x_token_int) send ARGC00000008
distcc[13039] (dcc_x_token_int) send ARGV00000003
distcc[13039] (dcc_x_token_string) send string 'gcc'
distcc[13039] (dcc_x_token_int) send ARGV00000008
distcc[13039] (dcc_x_token_string) send string '-std=c99'
distcc[13039] (dcc_x_token_int) send ARGV00000005
distcc[13039] (dcc_x_token_string) send string '-Wall'
distcc[13039] (dcc_x_token_int) send ARGV0000001d
distcc[13039] (dcc_x_token_string) send string '@_out/Release/Boot/Main.o.via'
distcc[13039] (dcc_x_token_int) send ARGV00000002
distcc[13039] (dcc_x_token_string) send string '-c'
distcc[13039] (dcc_x_token_int) send ARGV0000002f
distcc[13039] (dcc_x_token_string) send string '/home/user/sandbox/distcc-makefiles/Boot/Main.c'
distcc[13039] (dcc_x_token_int) send ARGV00000002
distcc[13039] (dcc_x_token_string) send string '-o'
distcc[13039] (dcc_x_token_int) send ARGV00000018
distcc[13039] (dcc_x_token_string) send string '_out/Release/Boot/Main.o'
distcc[13039] (dcc_note_state) note state 3, file "(NULL)", host "(NULL)"
distcc[13039] (dcc_collect_child) cpp child 13041 terminated with status 0
distcc[13039] (dcc_collect_child) cpp times: user 0.000000s, system 0.000000s, 0 minflt, 0 majflt
distcc[13039] cpp /home/user/sandbox/distcc-makefiles/Boot/Main.c on localhost completed ok

... {{ SECTION D }}

distcc[13039] (dcc_unlock) release lock fd4
distcc[13039] (dcc_x_file) send 14682 byte file /tmp/distcc_cd4d9216.i with token DOTI and compression 69
distcc[13039] (dcc_x_token_int) send DOTI0000395a
distcc[13039] (dcc_compile_remote) client finished sending request to server
distcc[13039] (dcc_note_state) note state 5, file "(NULL)", host "172.17.0.2"
distcc[13039] (dcc_select_for_read) select for read on fd6 for 300s
distcc[13039] (dcc_r_token_int) got DONE00000001
distcc[13039] (dcc_note_state) note state 6, file "(NULL)", host "(NULL)"
distcc[13039] (dcc_r_token_int) got STAT00000100
distcc[13039] (dcc_r_token_int) got SERR00000044
distcc[13039] (dcc_r_file) received 68 bytes to file /tmp/distcc_server_stderr_cd009216.txt
distcc[13039] (dcc_r_token_int) got SOUT00000000
distcc[13039] (dcc_r_token_int) got DOTO00000000

... {{ SECTION E }}

distcc[13039] 14682 bytes from /home/user/sandbox/distcc-makefiles/Boot/Main.c compiled on 172.17.0.2 in 0.0063s, rate 2274kB/s
distcc[13039] (dcc_unlock) release lock fd3
distcc[13039] ERROR: compile /home/user/sandbox/distcc-makefiles/Boot/Main.c on 172.17.0.2/1 failed
distcc[13039] (dcc_build_somewhere) Warning: remote compilation of '/home/user/sandbox/distcc-makefiles/Boot/Main.c' failed, retrying locally
distcc[13039] (dcc_mark_timefile) mark /home/user/.distcc/lock/backoff_tcp_172.17.0.2_3632_0
distcc[13039] Warning: failed to distribute /home/user/sandbox/distcc-makefiles/Boot/Main.c to 172.17.0.2/1, running locally instead

where, to my understanding, SECTION A finds the remote server, SECTION B opens a connection to the server, SECTION C send the command to run in the server (includes the compile options and the file to compile), SECTION D fetches the result, and SECTION E analizes the result and decides what to do next. As you might see, this is the error:

distcc[13039] (dcc_x_token_string) send string '@_out/Release/Boot/Main.o.via'
distcc[13039] (dcc_x_token_int) send ARGV00000002

Answer 2

To see output of compile error, define these environment above:

export DISTCC_VERBOSE=1
export DISTCC_SAVE_TEMPS=1
export DISTCC_FALLBACK=0

see also man distcc or https://linux.die.net/man/1/distcc

Answer 3

current distcc maintainer,

I highly recommend using MASQUERADE as documented in the distcc man page.