Coverage analysis of some googletest macros shows incomplete coverage when spread over several lines - why?

Question

I am using gcov/lcov for coverage analysis of googletest unit tests.

One recurring problem is that the coverage report shows uncovered lines in the test code for some googletest macros when the macro is spread over several lines.

I know gov/lcov cannot be more accurate than a single line, but I am puzzled by the behaviour that I see. Can someone explain this? Minimal example:

#include <gtest/gtest.h>

TEST(coverage,incomplete)
{
  // Every second line in every invocation here will show up as uncovered:
  EXPECT_NO_THROW(40 +
                  2);
  EXPECT_NO_THROW(40 + 2
                  );
  EXPECT_NO_THROW(40 + 2)
    ;
}

TEST(coverage,complete)
{
  // This test does not show uncovered lines
  EXPECT_NO_THROW(40 + 2);
  EXPECT_EQ(40
            +
            2
            , // even though this is spread over several lines
            42
            )
    ;
}

How coverage analysis was done:

g++-4.8 -Igtest/googletest/include/ --coverage -o coverage_macropp coverage_macropp.cpp gtest/googletest/make/gtest_main.a -pthread
./coverage_macropp
lcov --capture --directory . --output-file coverage.info
genhtml --demangle-cpp coverage.info --output-directory coverage

The coverage analysis in the web browser will then show lines 7, 9, and 11 as uncovered:

      Line data    Source code

   1             : #include <gtest/gtest.h>
   2             : 
   3           5 : TEST(coverage,incomplete)
   4             : {
   5             :   // Every second line in every invocation here will show up as uncovered:
   6           1 :   EXPECT_NO_THROW(40 +
   7           0 :                   2);
   8           1 :   EXPECT_NO_THROW(40 + 2
   9           0 :                   );
  10           1 :   EXPECT_NO_THROW(40 + 2)
  11           0 :     ;
  12           1 : }
  13             : 
  14           5 : TEST(coverage,complete)
  15             : {
  16             :   // This test does not show uncovered lines
  17           1 :   EXPECT_NO_THROW(40 + 2);
  18           1 :   EXPECT_EQ(40
  19             :             +
  20             :             2
  21             :             , // even though this is spread over several lines
  22             :             42
  23             :             )
  24           1 :     ;
  25           4 : }

Why? And why is the EXPECT_EQ macro not affected?

Answer 1

This was a fascinating investigation. Looking at these macros, I have learned things about switch/case and goto, that I did not know were possible.

But the reason for the behaviour differences come from if/else constructs, which have in common that the else branches are never executed, but which differ in the fact whether the compiler already knows, that these else branches will never be executed.

First, spreading the macro across multiple lines in the source code has the preprocessor produce (as far as it matters for this question) the following code for the compiler and its coverage analyzer:

if (condition) statement1; else statement2


  ;

And apparently, the coverage analyzer gcov, counts the line with the lonely semicolon as a code line, and considers it executed when the statement2 from the else branch is executed.

Now, to reproduce the coverage analysis difference, as observed in the question, consider this example program:

#include <stdio.h>
#include <time.h>

int main(int, char*[]) {
  const bool true_1 = true;
  const bool true_2 = time(NULL) != 0;

  if (true_1) 42; else printf("hello\n")
                    ;
  if (true_2) 42; else printf("hello\n")
                    ;
  return 0;
}

true_1 and true_2 are always true (in my lifetime, and if I do not mess with computers' clocks), but in the case of true_1, the compiler knows it, while for true_2, it can not know. (Yes, I could probably find a safer initializer for true_2. This will do for now.)

Note also that the statements in the if-branches do nothing, but the statements in the else-branches would have side effects.

The gcov/lcov coverage analysis for this program looks like this:

      Line data    Source code

   1             : #include <stdio.h>
   2             : #include <time.h>
   3             : 
   4           1 : int main(int, char*[]) {
   5           1 :   const bool true_1 = true;
   6           1 :   const bool true_2 = time(NULL) != 0;
   7             :   
   8             :   if (true_1) 42; else printf("hello\n")
   9             :                     ;
  10           1 :   if (true_2) 42; else printf("hello\n")
  11           0 :                     ;
  12           1 :   return 0;
  13             : }

Because the compiler already knows that true_1 is true, the else branch, and therefore, line 9, is not considered for coverage analysis. This is even true when compiling with -O0.

Line 11 in this program, however, is considered for coverage analysis, because true_2 is only known at runtime.

As a side note: if I had used another dummy-statement also in the else branch, or even a function call that is known to have no side effect, like sin(7) instead of printf("hello"), then this line would also not count in gcov coverage analysis.

So far it looks that, for complete formal coverage, I have to constrain googletest macros to a single source line, if the test result is not already known at compile time.