How to turn makefile into JSON Compilation Database?

Question

Say we have make files (not cmake/premake/ninja etc) for our project that do work for gcc and clang. We want to generate out from them JSON Compilation Database to feed it into clang-modernize tool. How to do such thing? (is there any parser in clang infrastructure or some script with usage like make CC='cc_args.py gcc' CXX='cc_args.py g++' or some other tool)?

Answer 1

I have no personal experience with it but Bear seems to be targeted to your scenario. (It was linked from the clang-modernize site.)

Answer 2

Make has a usually-undesired feature of emitting the compiler command lines to its standard output, but in this case you can use it with a shell (and jq) pipeline. With GNU Make on Bash:

make --always-make --dry-run \
 | grep -wE 'gcc|g\+\+' \
 | grep -w '\-c' \
 | jq -nR '[inputs|{directory:".", command:., file: match(" [^ ]+$").string[1:]}]' \
 > compile_commands.json

On Windows, cmd.exe syntax is similar:

• use ^ instead of \ for line continuation
• use "-delimited strings only
• use """ to escape a " within a "-delimited string
• use ^^ to escape a ^ within a "-delimited string

However, this doesn't handle cases where the makefile recursively invokes itself (or other makefiles) in subdirectories using the -C argument.

Answer 3

Maybe someone is in my same situation. I have a macOS and Bear was generating an empty compile_commands.json file. They acknowledge this problem in the README and suggest to use scan-build as a workaround. Unfortunately, I was still getting an empty file. Two alternative solutions I found are:

• compiledb, which worked in my case.
• a combination of clang++ compiler's -MJ option plus sed, detailed here, that seems simple enough and easy to transform into a make target. This one I have not tested.

Answer 4

If you don't have the permission to install Bear/compiledb, or you can't build Bear from source, then a Web app could help.

I built such a simple web app at https://texttoolkit.com/compilation-database-generator, which generates compile_commands.json from the make output.

Answer 5

Since I don't have enough points to react under answers yet, I wanted to comment under @Tanz87 's answer, who suggests to dry-run make, and uses grep and jq to format the output.

My CMake has detected /usr/bin/c++ as the default compiler (I'm on WSL2, Ubuntu 21.10), and thus the grep -wE 'gcc|g\+\+' command does not work, and just like @Hi-Angel, my compile_commands.json ended up with [].

The fix is to also grep for c++, so the whole command becomes:

make --always-make --dry-run \
 | grep -wE 'gcc|g\+\+|c\+\+' \
 | grep -w '\-c' \
 | jq -nR '[inputs|{directory:".", command:., file: match(" [^ ]+$").string[1:]}]' \
 > compile_commands.json

Edit: That being said, while the solution does seem to generate a compile_commands.json file, running clang-tidy in the build directory prints an error from the clang backend for every file in the compilation database:

error: no such file or directory: '&&' [clang-diagnostic-error]

Every command entry in the compile_commands.json file looks like this:

"cd /build/src && /usr/bin/c++  -I/src/src -std=gnu++2a -o CMakeFiles/project.dir/subdir/subsubdir/sourcefile.cpp.o -c /project/src/subdir/subsubdir/sourcefile.cpp

This project uses CMake to generate the Make build system files. Unfortunately, no solution yet.

Edit: Edit: Removing cd /build/src && from each command in the compile_commands.json fixes that particular error. Using sed:

sed 's|cd.*.&&||g' compile_commands.json

Edit: Edit: Edit: As a oneliner, it requires some extra slashes for the && tokens:

make --always-make --dry-run
 | grep -wE 'gcc|g\+\+|c\+\+'
 | grep -w '\-c'
 | sed 's|cd.*.\&\&||g'
 | jq -nR '[inputs|{directory:".", command:., file: match(" [^ ]+$").string[1:]}]' \
 > compile_commands.json

Answer 6

I put together this compiler wrapper, you may want to massage the if clause which extracts the filename argument. It only detects .c files

I invoke it like this: make ... CC="cc_wrapper $CC"

or: make ... CC="cc_wrapper gcc"

#! /bin/bash

# Writing to a common output file, so buffer for atomic output
if file=($(printf "%s\n" "$@" | grep '\.c$'))
then JSON="$(jq -nR '[inputs|{directory:$pwd, arguments:[$ARGS.positional[]], file:.}]' --arg pwd "$PWD" --args -- "$@" <<< "${file}")"
     # c2rust transpile "$file.compile_commands.json" will still look for a file called compile_commands.json so we have to provide it
     mkdir -p "$file.c2rs"
     echo "$JSON" > "$file.c2rs/compile_commands.json"
fi
exec "$@"

Answer 7

I have found that for some make projects compiledb is able to intercept make output and generate a valid compile_commands.json file. However, for some projects it generates an empty compile_commands.json file. The solution that always works for me is as below.

make clean;make VERBOSE=y all &> make_output.txt
compiledb --parse make_output.txt

This always generates a valid compile_commands.json file even for make projects where compiledb is not able to directly intercept the make output.
Hope this helps.