Catch test case order

Question

Can I guarantee the order of execution with multiple TEST_CASEs with Catch? I am testing some code using LLVM, and they have some despicable global state that I need to explicitly initialize.

Right now I have one test case that's like this:

TEST_CASE("", "") {
    // Initialize really shitty LLVM global variables.
    llvm::InitializeAllTargets();
    llvm::InitializeAllTargetMCs();
    llvm::InitializeAllAsmPrinters();
    llvm::InitializeNativeTarget();
    llvm::InitializeAllAsmParsers();
    // Some per-test setup I can make into its own function
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile...));
    CHECK_NOTHROW(Interpret(...));
    CHECK_THROWS(Compile(...));
    CHECK_THROWS(Compile(...));
}

What I want is to refactor it into three TEST_CASE,

one for tests that should pass compilation,
one for tests that should fail, and
one for tests that should pass interpretation (and in the future, further such divisions, perhaps).

But I can't simply move the test contents into another TEST_CASE because if that TEST_CASE is called before the one that sets up the inconvenient globals, then they won't be initialized and the testing will spuriously fail.

I am not familiar with this syntax (`CHECK_THROWS` etc), but maybe the `boost::unit_test` ["manually registered" syntax](http://www.boost.org/doc/libs/1_54_0/libs/test/doc/html/utf/user-guide/test-organization/manual-test-case-template.html) can give you an idea. Or did I fail to understand something basic about the question? — anatolyg, Oct 30 '13 at 22:36
Well, the whole "It uses a completely different unit testing framework" thing does kinda prevent `boost::unit_test` from being applicable. — Puppy, Oct 30 '13 at 22:51
@DeadMG Which framework is that? Is it specific to llvm or general-purpose? (it probably won't help me answer your question, but maybe if you clarify this, you will get more chance of an answer) — anatolyg, Oct 30 '13 at 23:13
It's the Catch unit testing framework. As referenced in "Catch test case order" and "`TEST_CASE` with Catch". And some moron edited `catch` into `try-catch` which is plain wrong. — Puppy, Oct 31 '13 at 09:57
Ahem, please excuse my sarcasm. I thought this was a relatively easy question to grok. Of course, the name "Catch" is not very unambiguous. — Puppy, Oct 31 '13 at 21:44

score 4 · Answer 1 · answered May 26 '14 at 15:35

I'm a bit late to this as I only just saw it - sorry (in future you can post Catch related questions to the Catch forum or issues list on GitHub, if appropriate.

Anyway - I don't know what you did in the end but in this case it sounds like you just want to group each set of assertions into SECTIONs.

TEST_CASE() {
    // Initialize really shitty LLVM global variables.
    llvm::InitializeAllTargets();
    llvm::InitializeAllTargetMCs();
    llvm::InitializeAllAsmPrinters();
    llvm::InitializeNativeTarget();
    llvm::InitializeAllAsmParsers();

    SECTION( "should pass compilation" ) {
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile...));
    }
    SECTION( "should pass interpretation" ) {
      CHECK_NOTHROW(Interpret(...));
    }
    SECTION( "Should fail compilation" ) {
      CHECK_THROWS(Compile(...));
      CHECK_THROWS(Compile(...));
    }
}

Each section then acts like an embedded test case (the whole test case is executed from the start - through all the initialisation - for each section). So if one of the no-throws throws it will not prevent the other sections from executing.

... unless the initialisation code should only be executed once - in which case you could either put in a static initialiser, as @paddy suggested (a class that calls the initialisers in its constructor - then just create a global instance) - or you could protect the block of initialisation code with an if on a static bool.

score 2 · Answer 2 · edited Mar 15 '17 at 09:48

If Phil's solution doesn't suit for some reason, here's an alternative:

struct TestFixture {
  static bool _initialised;
  TestFixture() {
    if (!_initialised) {
      llvm::InitializeAllTargets();
      llvm::InitializeAllTargetMCs();
      llvm::InitializeAllAsmPrinters();
      llvm::InitializeNativeTarget();
      llvm::InitializeAllAsmParsers();
      _initialised = true;
    }
  }
};

bool TestFixture::_initialised = false;

TEST_CASE_METHOD(TestFixture, "should pass compilation" ) {
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
}

TEST_CASE_METHOD(TestFixture, "should pass interpretation" ) {
  CHECK_NOTHROW(Interpret(...));
}

TEST_CASE_METHOD(TestFixture, "Should fail compilation" ) {
  CHECK_THROWS(Compile(...));
  CHECK_THROWS(Compile(...));
}

In this sample code, it doesn't matter which TEST_CASE runs first because the first-run one will call the llvm initialisation functions and the others will skip this due to the bool.

This code makes use of Catch's Test Fixture support, which we use extensively in my day job: https://github.com/philsquared/Catch/blob/master/docs/test-fixtures.md

Catch test case order

2 Answers2

Linked