Catch.hpp unit testing: How to dynamically create test cases?

Question

I am using CATCH v1.1 build 14 to do unit testing of my C++ code.

As part of the testing, I would like to check the outputs of several modules in my code. There is not a set number of modules; more modules may be added at any time. However, the code to test each module is identical. Therefore, I think it would be ideal to put the testing code in a for loop. In fact, using catch.hpp, I have verified that I can dynamically create Sections within a Test Case, where each Section corresponds to a module. I can do this by enclosing the SECTION macro in a for loop, for example:

#include "catch.hpp"
#include <vector>
#include <string>
#include "myHeader.h"

TEST_CASE("Module testing", "[module]") {
    myNamespace::myManagerClass manager;
    std::vector<std::string> modList;
    size_t n;

    modList = manager.getModules();
    for (n = 0; n < modList.size(); n++) {
        SECTION(modList[n].c_str()) {
            REQUIRE(/*insert testing code here*/);
        }
    }
}

(This is not a complete working example, but you get the idea.)

Here is my dilemma. I would like to test the modules independently, such that if one module fails, it will continue testing the other modules instead of aborting the test. However, the way CATCH works, it will abort the entire Test Case if a single REQUIRE fails. For this reason, I would like to create a separate Test Case for each module, not just a separate Section. I tried putting my for loop outside the TEST_CASE macro, but this code fails to compile (as I expected):

#include "catch.hpp"
#include <vector>
#include <string>
#include "myHeader.h"

myNamespace::myManagerClass manager;
std::vector<std::string> modList;
size_t n;

modList = manager.getModules();
for (n = 0; n < modList.size(); n++) {
    TEST_CASE("Module testing", "[module]") {
        SECTION(modList[n].c_str()) {
            REQUIRE(/*insert testing code here*/);
        }
    }
}

It might be possible to do this by writing my own main(), but I can't see how to do it exactly. (Would I put my TEST_CASE code directly into the main()? What if I want to keep my TEST_CASE code in a different file? Also, would it affect my other, more standard Test Cases?)

I can also use CHECK macros instead of REQUIRE macros to avoid aborting the Test Case when a module fails, but then I get the opposite problem: It tries to continue the test on a module that should have failed out early on. If I could just put each module in its own Test Case, that should give me the ideal behavior.

Is there a simple way to dynamically create Test Cases in CATCH? If so, can you give me an example of how to do it? I read through the CATCH documentation and searched online, but I couldn't find any indication of how to do this.

Answer 1

There is a way to achieve what you're looking for, but I'd observe that you're going about this the wrong way:-

Unit tests are to meant to test each unit, i.e. you write a component and a test to verify the correct behaviour of that component. If you later decide to change one component in some way, you update the corresponding test.

If you aggregate all of your tests for all of your components into the same file, it becomes much harder to isolate the unit that behaves differently.

If you want to factor out the testing of a component because it's essentially the same across all your components, you could do one of the following:

1. Extract the common tests to a separate header file

You can #define the type name of the component you wish to test and then include a header file with all the tests in it:

// CommonTests.t.h
#include "catch.hpp"
TEST_CASE("Object Can be instantiated", "[ctor]")
{
   REQUIRE_NOTHROW(COMPONENT component);
}

// SimpleComponent.t.cpp
#define COMPONENT SimpleComponent
#include "CommonTests.t.h"

This is straightforward to do, but has a downside - when you run the test runner, you'll have duplicate tests (by name) so you can only run all tests, or by tag.

You can solve this by stringizing the component name and pre/appending it to the test case name.

** 2. Call common tests by parameterising the component **

Put your common tests into a separate file and call the common test methods directly:

// CommonTests.t.h
void RunCommonTests(ComponentInterface& itf);

// CommonTests.t.cpp
void RunCommonTests(ComponentInterface& itf)
{
  REQUIRE(itf.answerToLifeUniverseAndEverything() == 42);
}

// SimpleComponent.t.cpp
#include "SimpleComponent.h"
#include "CommonTest.t.h"
#include "catch.hpp"

TEST_CASE("SimpleComponent is default-constructible", "[ctor]")
{
   REQUIRE_NOTHROW(SimpleComponent sc);
}

TEST_CASE("SimpleComponent behaves like common components", "[common]")
{
  SimpleComponent sc;
  RunCommonTests(sc);
}

Answer 2

It sounds like Catch might be migrating toward property-based testing, which I'm hoping will allow a way to dynamically create test cases. In the meantime, here's what I ended up doing.

I created a .cpp file with a single TEST_CASE for a single module, and a global variable for the module name. (Yes, I know global variables are evil, which is why I am being careful and using it as the last resort):

module_unit_test.cpp:

#include "catch.hpp"
#include <string>
#include "myHeader.h"

extern const std::string g_ModuleName;  // global variable: module name

TEST_CASE("Module testing", "[module]") {
    myNamespace::myManagerClass manager;
    myNamespace::myModuleClass *pModule;
    SECTION(g_ModuleName.c_str()) {
        pModule = manager.createModule(g_ModuleName.c_str());
        REQUIRE(pModule != 0);
        /*insert more testing code here*/
    }
}

Then, I create an executable that will run this test on a single module specified on the command line. (I tried looping through the Catch::Session().run() below, but Catch does not allow it to run more than once.) The object file from the code below module_test.cpp and from the unit test code above module_unit_test.cpp are linked when creating the executable.

module_test.cpp:

#define CATCH_CONFIG_RUNNER
#include "catch.hpp"
#include <string>
#include <cstdio>

std::string g_ModuleName;  // global variable: module name

int main(int argc, char* argv[]) {
    // Make sure the user specified a module name.
    if (argc < 2) {
        std::cout << argv[0] << " <module name> <Catch options>" << std::endl;
        return 1;
    }

    size_t n;
    char* catch_argv[argc-1];
    int result;

    // Modify the input arguments for the Catch Session.
    // (Remove the module name, which is only used by this program.)
    catch_argv[0] = argv[0];
    for (n = 2; n < argc; n++) {
        catch_argv[n-1] = argv[n];
    }

    // Set the value of the global variable.
    g_ModuleName = argv[1];

    // Run the test with the modified command line arguments.
    result = Catch::Session().run(argc-1, catch_argv);

    return result;
}

Then, I do the looping in a separate executable (not linked to the object files from the code above):

module_test_all.cpp:

#include <cstdlib>
#include <vector>
#include <string>
#include "myHeader.h"

int main(int argc, char* argv[]) {
    std::string commandStr;
    int result, status = 0;
    myNamespace::myManagerClass manager;
    std::vector<std::string> modList;
    size_t m, n;

    // Scan for modules.
    modList = manager.getModules();

    // Loop through the module list.
    for (n = 0; n < modList.size(); n++) {
        // Build the command line.
        commandStr = "module_test " + modList[n];
        for (m = 1; m < argc; m++) {
            commandStr += " ";
            commandStr += argv[m];
        }

        // Do a system call to the first executable.
        result = system(commandStr.c_str());

        // If a test fails, I keep track of the status but continue
        // looping so all the modules get tested.
        status = status ? status : result;
    }

    return status;
}

Yes, it is ugly, but I have confirmed that it works.