Run tests in specific order

Question

I've installed in Visual Studio 2013 the Google Test runner extension.

Now I have a test project

TestProject
    |
    |-InitializationTests.cpp
    |-RunningTests.cpp

I want to run all test cases inside InitializationTests.cpp before tests inside RunnintTests.cpp. How can I accomplish this?

You probably saw this question coming, but are you *sure* you want your test order to matter, indicating your tests somehow depend on each other? — stijn, Jan 14 '15 at 08:23
I want to run running tests only if initialization tests are corrects. What's wrong with this? googletest allows `ASSERT_*` that stop other test running. What's the meaning of this if order does not matter? — Jepessen, Jan 14 '15 at 08:31
There's not that much wrong in not wanting to run tests when others fail as some sort of optimization in build time, but it doesn't really add any value at all - in the end you'll have to run all tests anyway and it just puts an extra maintainance burden on the code (as proven by the very fact you're asking this question). — stijn, Jan 14 '15 at 08:46
@Jepessen: your init tests will fail anyway. 'Running' tests will probably also do. You will have a report and you may start investigation of a problem from any tests group you want. — Michał Walenciak, Jan 14 '15 at 08:57
Sure, tests should be able to run in any order. But sometimes they can't, due to a bug. When that happens I want to fix this bug. In order to do that I need to identify the bug, and often the best way to do that is to force the tests to run in a particular order so that I create a minimal repro and then identify the pair (or more) of tests that interact badly. Once the bug is fixed I can then let the tests run in any order — Bruce Dawson, Apr 29 '21 at 16:58

score 20 · Answer 1 · edited Oct 19 '16 at 11:10

20

Sure You Can! It's a program not concrete.
main.cpp:

::testing::GTEST_FLAG(filter) = "tA.*";  
RUN_ALL_TESTS();  
::testing::GTEST_FLAG(filter) = "tB.*";  
RUN_ALL_TESTS();  
::testing::GTEST_FLAG(filter) = "tC.*";  
return RUN_ALL_TESTS();

It will run in the next order:

 tA.*,   
 tB.*,  
 tC.*,

edited Oct 19 '16 at 11:10

maazza

7,016
15
63
96

answered Oct 14 '16 at 13:34

jstar

847
9
9

1

I think this is a good answer to the question. It gets the job done with out the dogmatic lecture. We all get the it shouldn't matter the what order your tests are run, but sometimes life doesn't work out that way. I would recommend tracking the return status of RUN_ALL_TESTS() though. Shame gtest doesn't allow for this more cleanly. Is this recommended? No. Does it work? Yes. – JeffCharter Jul 11 '17 at 16:25
1

@jstar: problem with this solution is that you can miss some of the tests. – Michał Walenciak Sep 10 '18 at 10:42
@MichałWalenciak it can be solved eg. `::testing::GTEST_FLAG(filter) = "*.*"s + "-" + executed_filters; return RUN_ALL_TESTS();` according to http://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests . Anyway, there are problems with passed filters outside (argv). – amordo Jan 10 '23 at 14:25

score 2 · Answer 2 · answered Jan 14 '15 at 09:51

Test framework normally do not allow to control the order of tests, because it is a general requirement that tests are independant from each other.

But you can always run a single test, and Google Test has a powerful option to control which tests are to be run. From Google Test advanced guide : If you set the GTEST_FILTER environment variable or the --gtest_filter flag to a filter string, Google Test will only run the tests whose full names (in the form of TestCaseName.TestName) match the filter

For your use case, supposing you execute all tests in your test project by calling :

TestProject

you could run only initialization tests by running :

TestProject --gtest_filter=InitializationTests.*

(provided InitialisationTests.cpp contains tests for test case InitializationTests)

score -2 · Answer 3 · answered Jan 14 '15 at 08:25

-2

Tests order should be independent. And GTest does not allow to control the order. Even more: tests can be run in parallel

answered Jan 14 '15 at 08:25

Michał Walenciak

4,257
4
33
61

Ok but I've a test in which I say "check if this operation throws something", and then I write a test in which I use the previously tested operation. Google Test has also `ASSERT_*` that stop further execution. What's the meaning of this if order cannot be defined? – Jepessen Jan 14 '15 at 08:27
1

@Jepessen: In one test ( `TEST(foo, bar)`) you may have many `ASSERT`s and `EXPECT`s. These are checked in order as code in the test is executed in order. If you check for more than one thing, you may want to break execution of `TEST` by using `ASSERT` – Michał Walenciak Jan 14 '15 at 08:30
But if I want to check more things, I need to use more unit tests, or I'm misunderstanding their use? – Jepessen Jan 14 '15 at 08:36
@Jepessen: `TEST` should check one functionality, but may do it in more than one way. For example You may have a function `int add(int, int)` and you may want to have a: `TEST(AddTest, addsPositiveIntegers)` with: `EXPECT_EQ(2, add(1,1)); EXPECT(128, add(64,64)); ...` then you may have a test for negative numbers, for mix, a check for overflows etc etc – Michał Walenciak Jan 14 '15 at 08:53
1

How about debugging test dependencies that only show up when they're run in a specific order? – rphv Jan 12 '21 at 23:19

Run tests in specific order

3 Answers3