Testing Framework that supports Ordering of Test Classes and Test Methods

Question

First of all: I don't want to do unit tests, instead I am going to implement API tests. These tests should be executed in a defined order:

Login
Get something
Create something
Logout

I'd like to get this work in Visual Studio as well as by SonarQube (Gallio is the test automation platform I am using).

My problem is just to find a framework that supports my requirements.

I've already tested:

MSTest -> Does not support ordered tests at all, except implementing the Ordered Test template that is not supported by any Gallio test runner plugin.
XUnit -> It is possible to implement an ordering for test methods within test classes, but it is not possible to order test classes.

Is there any testing framework that supports this requirement and provides runner for Visual Studio as well as an appropriate plugin for Gallio?

score 1 · Accepted Answer · answered Aug 01 '13 at 18:19

1

MSTest has a TestInitialize attribute that you can use to initialize each test. In it you could put your Login. There is also a TestCleanup attribute. There you could put your LogOut. The Get and Create would have to be within one method in order to ensure that the Get always happens before the Create.

I am not familiar with SonarQube, so don't know whether it would handle this.

answered Aug 01 '13 at 18:19

DeborahK

57,520
12
104
129

That is not enough. As I want to implement API tests there are several steps to do. Login/logout is just an example. I need an ordering based on test classes and tests inside those classes. Therefore TestInitialize and TestCleanup do not really help. But thanks for your answer. – LaOsgaar Aug 02 '13 at 06:48
I ended up using xUnit as the Framework by implementing a priority attribute for ordering within a test fixture. All really necessary things are done by the constructor of my base class (this is equivalent to TestInitialize of MSTest). The advantage of using XUnit is that all tests (needs the XUnit runner installed) can be executed within Visual Studio (based on MSTest this can't be done, except filtering tests and working with that very ugly ordered test definition file). – LaOsgaar Aug 02 '13 at 11:13

score 0 · Answer 2 · answered Aug 01 '13 at 16:55

0

The ones you specified don't support this because they are all unit test frameworks. Unit tests are supposed to be testing "units" of code, and therefore are independent of other tests. What you are looking for is integration tests or acceptance tests.

I haven't really done integration / acceptance testing so I can't suggest what you should use, but you can google around for some. The only one I know of off the top of my head is StoryTeller

answered Aug 01 '13 at 16:55

Darren Kopp

76,581
9
79
93

Yeah, I know that the frameworks I mentioned are unit test Frameworks. Therefore I understand why they act like they do. But I'd like to have the API tests integrated into Visual Studio as well as being able to execute them by SonarQube and get the results analyed, also by SonarQube. At the Moment I do not see another option. – LaOsgaar Aug 02 '13 at 06:54

Testing Framework that supports Ordering of Test Classes and Test Methods

2 Answers2