TEST_P tests in library do not run when INSTANTIATE_TEST_CASE_P is called in a separate executable

Question

I'm trying to follow the instructions for Creating Value-Parameterized Abstract Tests in the googletest README. I've created a project directory, in which I have CMakeLists.txt, fixture.hh, fixture.cc, test.cc, and the entire Google Test repository checked out within subdirectory googletest. My goal is simply to create a library with a text fixture class and a set of TEST_P tests that can be linked against by individual unit test executables to minimize code duplication.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.10 FATAL_ERROR)

enable_testing()
add_subdirectory(googletest)
include(GoogleTest)

add_library(test_utils STATIC fixture.cc)
target_link_libraries(test_utils PUBLIC gtest)

add_executable(unit_tests test.cc)
target_link_libraries(unit_tests PRIVATE test_utils gtest_main)
gtest_discover_tests(unit_tests)

// fixture.hh

#pragma once

#include <gtest/gtest.h>

namespace test_utils { /* *************************************************** */

class Fixture : public ::testing::TestWithParam<size_t> {};

} /* namespace test_utils *************************************************** */

// fixture.cc

#include "fixture.hh"

using namespace test_utils;

TEST_P( Fixture, foo )
{
  ASSERT_EQ( GetParam()%2, 0 );
}

// test.cc

#include "fixture.hh"

namespace { /* ************************************************************** */

using namespace test_utils;

INSTANTIATE_TEST_CASE_P( BarInstantiation, Fixture, ::testing::Values( 18 ) );

} /* namespace ************************************************************** */

When I run ./unit_tests, it outputs:

Running main() from ../googletest/googletest/src/gtest_main.cc
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

ctest outputs No tests were found!!!. What am I doing wrong?

Answer 1

As you know, the standard procedure for unit-testing with googletest is:

1. Build a library containing the code-under-test.
2. Write your googletest unit-tests in one or more source files.
3. Optionally, write a source file that defines main in the standard googletest manner to run the unit-tests. Alternatively, you may not define any main but instead add libgtest_main to your linkage.
4. Compile all the googletest sources from 2 and 3 to object files.
5. Link a googletest test-runner, inputting all object files together with the library from 1 plus libgtest_main (if you need it), plus libgtest

You would like to link libgtest_main instead of defining main, and you would like to vary the procedure in an unorthodox way by archiving all the object files from 2 in a static library and adding it to your linkage instead of linking the object files directly.

And in your posted example, there's a further trivial variation. You haven't got any library of code-under-test, because it's not needed to illustrate your problem. You've just got a place-holder unit-test that is bound to pass, if it gets run. The problem being that it doesn't even get run!

Let's put CMake aside for the moment as it's an irrelevant distraction for explaining your puzzle. Here I have your files as posted:

$ ls -R
.:
fixture.cc  fixture.hh  test.cc

First I'm going to build a googletest runner by the standard procedure (except for the fact that there's no code-under-test).

Compile the sources:

$ g++ -Wall -Wextra -c test.cc fixture.cc

Link:

$ g++ -o tester test.o fixture.o -lgtest_main -lgtest -pthread

Run:

$ ./tester
Running main() from /home/imk/Downloads/googletest-master/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BarInstantiation/Fixture
[ RUN      ] BarInstantiation/Fixture.foo/0
[       OK ] BarInstantiation/Fixture.foo/0 (0 ms)
[----------] 1 test from BarInstantiation/Fixture (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (0 ms total)
[  PASSED  ] 1 test.

Nothing wrong with that.

Now I'll make the test runner again the way you want to. No need to recompile any sources. I'll just make a static library contaning fixture.o:

$ ar rcs libutest.a fixture.o

And relink using it instead of fixture.o:

$ g++ -o tester test.o -L. -lutest -lgtest_main -lgtest -pthread

And again run:

$ ./tester
Running main() from /home/imk/Downloads/googletest-master/googletest/src/gtest_main.cc
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

There's your puzzle. There are no tests now. How come?

Finally I'm going to link the test runner a third way:

$ g++ -o tester test.o -lgtest_main -lgtest -pthread

Not linking either fixture.o or libutest.a. And that one runs:

$ ./tester
Running main() from /home/imk/Downloads/googletest-master/googletest/src/gtest_main.cc
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.

exactly like the second one. In this case that's not surprising. We know that a googletest runner can be linked without any tests, and that's what it will do.

Let's do some study of the symbols that are defined in fixture.o.

$ nm -C --defined-only fixture.o | egrep -w '(W|V)' | wc -l
463

There 463 weak symbols defined. And there are just 2 strong symbols defined, which are:

$ nm -C --defined-only fixture.o | egrep -w '(A|B|C|D|G|R|S|T)'
0000000000000000 B Fixture_foo_Test::gtest_registering_dummy_
0000000000000000 T Fixture_foo_Test::TestBody()

The linker distinguishes importantly between weak and strong symbols.

The linker will not link a program that contains any strong symbol reference that it fails to resolve, i.e. for which it can find no definition in the linkage. It will fail the linkage with undefined reference errors. But it does not have to resolve weak references to link a program. An unresolved reference to a weak symbol simply remains null ( = 0) in the program.

The linker will not link a program (or shared library) whose linkage produces multiple strong definitions of the same symbol. It will fail the linkage with multiple definition errors. But it will tolerate multiple weak definitions as well as one strong definition of the same symbol, in which case it picks the strong definition to resolve references and discards all the weak ones. And it will tolerate multiple weak definitions of a symbol without any strong one. In that case it will pick one of the weak definitions arbitrarily - in practice, just the first one that it sees - and discards all the rest.

A C++ compiler routinely emits mixtures of weak and strong symbol definitions in order to translate C++ into successfully linkable object code. Inline class methods and instantiations of function templates have to be compiled with weak definitions.

The linker's different obligations to weak references and strong references are connected with its different obligations to an input file that is object file and one that is a static library.

When you offer an object file foo.o to the linker, it will link it into the program unconditionally.

A static library is an archive of object files that you offer to the linker from which to extract just the ones it needs, and link them into the program. By default it won't extract and link any of them unconditionally. Only the ones that resolve symbols already referred to in the linkage for which it needs to find definitions.

But a weak symbol reference is one that the linker does not need to resolve to link a program. So by default it will never extract and link an object file from a static library to resolve a weak symbol reference. The reference is allowed to be undefined in the program, so the linker will expend no effort searching libraries to define it.

Putting together the facts we've just reviewed, we know that the linkage:

$ g++ -o tester test.o fixture.o -lgtest_main -lgtest -pthread

which executed the TEST_P defined in fixture.cc, is one that will link fixture.o into the program, because it is an object file.

And we also know that the linkage:

$ g++ -o tester test.o -L. -lutest -lgtest_main -lgtest -pthread

which executed 0 tests, will extract and link libutest.a(fixture.o) into the program only if test.o contains at least one reference to a strong symbol that is defined in fixture.o, i.e. a reference to one of:

Fixture_foo_Test::gtest_registering_dummy_
Fixture_foo_Test::TestBody()

since all the 463 other symbols defined in fixture.o are weak.

But test.o makes no references to any symbols containing Fixture_foo_Test:

$ nm -C test.o | grep Fixture_foo_Test; echo Done
Done

So we can conclude that the linkage:

$ g++ -o tester test.o -L. -lutest -lgtest_main -lgtest -pthread

will not extract and link libutest.a(fixture.o). It is exactly the same linkage as:

$ g++ -o tester test.o -lgtest_main -lgtest -pthread

which as we know contains 0 tests.

Putting fixture.o into a static library renders it unnecessary to the linkage of tester, and it isn't linked.

In that light of all that, your CMakeLists.txt is easy to fix by simplifying it to:

cmake_minimum_required(VERSION 3.10 FATAL_ERROR)

enable_testing()
add_subdirectory(googletest)
include(GoogleTest)

add_executable(unit_tests test.cc fixture.cc)
target_link_libraries(unit_tests PRIVATE gtest_main gtest)
gtest_discover_tests(unit_tests)

which will result in fixture.o being unconditionally linked into unit_tests and the program will output:

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from BarInstantiation/Fixture
[ RUN      ] BarInstantiation/Fixture.foo/0
[       OK ] BarInstantiation/Fixture.foo/0 (0 ms)
[----------] 1 test from BarInstantiation/Fixture (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[  PASSED  ] 1 test.

as you hoped.

Clearly this solution doesn't satisfy your stated motivation for creating libtest_utils.a in the first place:

My goal is simply to create a library with a text fixture class and a set of TEST_P tests that can be linked against by individual unit test executables to minimize code duplication.

But that goal is misconceived. There is no point in building a static library from a set of TEST_P test sources (or any sources) if you are going to rebuild it from the same sources for linkage with each executable that needs some of the object files you will archive in the static library. If you were going to rebuild it from source for each executable that needs it, you would minimize code duplication just by keeping all the TEST_P test sources in one shared place or repository, and there would still be no point in rebuilding the static library for each executable. Each of those executable projects might just as well include some or all of the library's TEST_P sources among it's own sources and not bother rebuilding the library or linking against it.

On the other hand, you would have a code reuse motivation for building a static library from TEST_P test sources if you intended not to rebuild it for every unit-test executable project, but to build it as an independent project and simply consume it in the linkage of unit-test executables that you build later. You don't want to do that however, for the same good reasons that googletest itself is best built within each unit-test project that uses it, just as you are doing. It defeats the purpose of building googletest as an in-project dependency if you then link the executable with a library of object files that were compiled sometime in the past in a different project with a possibly different googletest revision, possibly with a different compiler.

You want googletest to be built by your unit-test project, and your TEST_P sources to be compiled and linked with that same googletest, in the same project. So there is no point in archiving the TEST_P object files that you build into intermediate static library, as opposed to simply linking them directly in the unit-test executable. And, if you do put them in an intermediate static library, by default the linker will ignore them.

I said by default the linker will ignore them, because it possible to force the linker to link an object file from within a static library even if it doesn't need to. You can do that by forcing it to extract and link all the object files in the static library, using the linker's --whole-archive option. If invoking the linker as usual via gcc/g++ (or clang/clang++), the linkage options you'd need to do this for a static library libbar.a would be:

... -Wl,--whole-archive -lbar -Wl,--no-whole-archive ...

You need to enable --whole-archive before -lbar and disable it afterwards because otherwise it will apply not only to -lbar but to all subsequent libraries, including even the the ones that you don't even mention in your commandline that are linked by default.

If you force the linkage of all object files in libbar.a like that, then of course any weak definitions that they contain will be used to resolve weak symbol references in the program.

However, it is not straightforward to achieve the linkage options -Wl,--whole-archive -lbar -Wl,--no-whole-archive in a CMake project that builds libbar.a. See this question and answer. And once again, there is no point in it, so why fight the framework? Googletest expects you to build your unit-tests in your unit-test project and do the obvious thing: link the object files with the program. If you do that, your unit-tests will be linked and run as you expect.

Answer 2

TEST_P is a macro which, when expanded, declares a full c++ class, onto which INSTANTIATE_TEST_CASE_P adds functionality.

Since your TEST_P is in a separate .cc file, your INSTANTIATE_TEST_CASE_P call (which is in its own .cc file) can't see it, so it's essentially defining a function with no owning class.

On the flip side, the class defined by TEST_P doesn’t get equipped with the method(s) INSTANTIATE_TEST_CASE_P would have defined for it. So it doesn’t find any test instantiations to run.

You need to move your TEST_P declaration to where your INSTANTIATE_TEST_CASE_P call can see it for it to work.

A caveat, though - TEST_P not only declares a class but defines several functions. So if you try to place it in a header file, you might receive some linker errors pertaining to these class methods (as already defined, for example). So the best place for it just might be right next to the INSTANTIATE_TEST_CASE_P call.

TEST_P is really meant to be its own independent test case, and it seems like it was designed to go into its own source file, and not accessible as a sort of library component, as it seems you're trying to do.

Answer 3

Could you check if it makes a difference if you link fixture.cc directly (without creating a static library first).

There is a test case in the official googletest repository that explicitly tests that test cases can be defined and instantiated in different translation units. I am not entirely sure, but adding a static library around fixture.cc it might lead to a static initialization ordering problem.

Answer 4

Going off of the description by Mike Kinghan, you can use CMake Object Libraries since CMake 3.12:

add_library(test_utils OBJECT fixture.cc)

Answer 5

We encountered this issue in our project. Based on discussion within Mike Kinghan's solution, You can try below solution if you want to use a static library contained the test body.

First, defining a function in your test class to generate the test case name as below codes showing.

// fixture.hh

#pragma once

#include <gtest/gtest.h>

namespace test_utils { /* *************************************************** */

class Fixture : public ::testing::TestWithParam<size_t> {

public:
       public:
       static std::string getTestCaseName(testing::TestParamInfo<size_t> obj){
            std::ostringstream name;
            name << "my_test_case_name_with_parameter_" << obj;
            return name.str();
       }
 
};

} /* namespace test_utils *************************************************** */

And then, use this function getTestCaseName() in INSTANTIATE_TEST_CASE_P as the last inputting parameter.

// test.cc

#include "fixture.hh"

namespace { /* ************************************************************** */

using namespace test_utils;

INSTANTIATE_TEST_CASE_P( BarInstantiation, 
                         Fixture, 
                         ::testing::Values( 18 )，
                         Fixture::getTestCaseName);

} /* namespace ************************************************************** */

Finally rebuild your project and will get the expected result.