How do I correctly handle output directories for GCC test coverage profiling?

Question

I am trying to generate coverage data for my unit tests, with the ultimate goal of displaying said data as part of our automated build output.

We build using makefiles and cs-make, with cxxtest as the unit test framework. The autogenerated test runner file is C++, with all our application files in C.

The unit test target and recipe for a unit test executable is:

.SECONDEXPANSION:
UnitTest/%.exe: UnitTest/Tests/$$*/$$*Tests_cxx.cpp
    cs-rm *.gcda
    ..\cxxtest\bin\cxxtestgen --error-printer -o $@ $(addsuffix .h, $(basename $@))
    gcc -ftest-coverage -fprofile-arcs $(INCLUDES) $($*_SOURCES) -o $@
    ./$@
    gcovr -v

With $(INCLUDES) and $(SOURCES) as appropriate.

The recipe is executed by running (for example)

cs-make cs-make -C TestPlatform -f UnitTest\makefile UnitTest\Test1.exe

This builds Test1.exe and generates .gcno files. Test1.exe runs and generates .gcda files. gcovr correctly interprets these files and outputs coverage data.

So far, so good. But we are actually building and running several unit tests in parallel by executing:

cs-make -C TestPlatform -j -f UnitTest\makefile UnitTest\all

where all is dependent on all defined unit test executables.

To support parallel builds, I added this option to my GCC call:

-fprofile-dir=UnitTest/Tests/$*/Coverage

which puts the .gcda files for each executable in a seperate directoy.

But now, gcovr does not process the files correctly:

gcovr -v
Filters for --root: (1)
- <_sre.SRE_Pattern object at 0x0000000004FED600>
Filters for --filter: (1)
- DirectoryPrefixFilter(C\:\/Workspace\/CoverageTest\/TestPlatform\/)
Filters for --exclude: (0)
Filters for --gcov-filter: (1)
- AlwaysMatchFilter()
Filters for --gcov-exclude: (0)
Filters for --exclude-directories: (0)
Scanning directory . for gcda/gcno files...
Found 6 files (and will process 4)
Pool started with 1 threads
Processing file: C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage\coverage.gcda
Running gcov: 'gcov C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage\coverage.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage' in 'c:\users\jfowkes\appdata\local\temp\tmpdfjja7'
Processing file: C:\Workspace\CoverageTest\TestPlatform\Test1Tests_cxx.gcda
Running gcov: 'gcov C:\Workspace\CoverageTest\TestPlatform\Test1Tests_cxx.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory C:\Workspace\CoverageTest\TestPlatform' in 'c:\users\jfowkes\appdata\local\temp\tmpdfjja7'
Processing file: C:\Workspace\CoverageTest\TestPlatform\coverage.gcda
Running gcov: 'gcov C:\Workspace\CoverageTest\TestPlatform\coverage.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory C:\Workspace\CoverageTest\TestPlatform' in 'c:\users\jfowkes\appdata\local\temp\tmpdfjja7'
Processing file: C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage\Test1Tests_cxx.gcda
Running gcov: 'gcov C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage\Test1Tests_cxx.gcda --branch-counts --branch-probabilities --preserve-paths --object-directory C:\Workspace\CoverageTest\TestPlatform\UnitTest\Tests\Test1\Coverage' in 'c:\users\jfowkes\appdata\local\temp\tmpdfjja7'
Gathered coveraged data for 0 files
------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
------------------------------------------------------------------------------
TOTAL                                          0       0    --%
------------------------------------------------------------------------------
cs-make: Leaving directory `C:/Workspace/CoverageTest/TestPlatform'

gcovr seems to be finding the files and handing them off to gcov, but no coverage data comes out.

I've tried various different gcovr options, but none seem to make this any better.

I'm running this build on Windows 10 in Eclipse CDT - this is requirement for us. I suspect based on other answers that there might be path and/or directory separator issues, but really I'm not sure of anything at the moment.

My question is: what changes do I need to make to by makefile so I can add this coverage data to our build, which continuing to support parallel builds?

I can of course restructure our unit testing to have a makefile-per-unit-test-suite, but this is substantial work and I would like to avoid it if at all possible.

Answer 1

The problem is gcovr/gcov need to correlate the coverage data with source files in order to generate the coverage report, and the use of -fprofile-dir makes this difficult. AFAIK gcovr is not currently able to handle that case correctly. Gcovr's --object-directory option is unable to help in this case.

Consider to cd into a temp directory where you perform the build. So roughly, moving from

gcc --coverage test/source.c -o testfoo
./testfoo
gcovr

to

[[ -d build-foo ]] || mkdir build-foo
cd build-foo && gcc --coverage ../test/source.c -o testfoo
cd build-foo && ./testfoo
cd build-foo && gcovr -r ../test

Unfortunately, this would be a variation of your “Makefile per test suite” approach.

Another alternative might be to run tests in parallel in other directories, and then copy the coverage data one test suite at a time back into your build directory. This requires that you provide appropriate paths for the GCOV_PREFIX environment variable. This would be the same procedure as for using gcovr in a cross-compilation environment, as explained by gocarlos on the gcovr issue tracker: https://github.com/gcovr/gcovr/issues/259#issuecomment-406788661