25

I need to run tests in order. I fail to find this adequately documented anywhere. I would prefer to do this from command line. Something like

 mvn -Dtest=test1,test2,test3,test5 test

How do I do this?

niken
  • 2,499
  • 3
  • 33
  • 56

6 Answers6

46

You can't specify the run order of your tests.

A workaround to do this is to set the runOrder parameter to alphabetical.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <runOrder>alphabetical</runOrder>
    </configuration>
</plugin>

and then you need to have rename your tests to obtain the expected order.

However it isn't a good idea to have dependent tests. Unit tests must be fIrst.

Leponzo
  • 624
  • 1
  • 8
  • 20
gontard
  • 28,720
  • 11
  • 94
  • 117
  • 6
    Which a really bad idea, cause unit tests should not depend on each other which is the case if you need an order of unit tests. – khmarbaise Aug 22 '12 at 15:12
  • 1
    @khmarbaise i totally agree with you. – gontard Aug 22 '12 at 15:24
  • 2
    @gontard I thought ur answer said "I totally disagree with you" - and I do. Tests should be independent, yes. BUT, there are times when you cannot mock parts of your system. Or need a more realistic interaction between them. You will end up breaking your head around how you should structure your unit tests, instead of getting the system up and running. – niken Aug 24 '12 at 20:47
  • 3
    @Nik "you cannot mock parts" sounds to me you that you are not writing unit tests but integration tests. If your code is not unit-testable, i will suggest you to write the test first (the best way to write testable code). But may be you can't because you are working with existing code. At last for me the integration tests, like the unit ones, are much better if they are isolated. – gontard Aug 25 '12 at 05:56
  • 1
    @gontard I agree 100% with your last comment! – Pacane Aug 25 '12 at 14:48
  • @gontard I never really thought about it that way, but I guess you're right. What you call integration testing, to me, is just a structured execution of unit tests. Kill 2 birds with 1 stone. – niken Aug 27 '12 at 20:43
  • 3
    this is in fact very useful when trying to find the culprit in a set of tests that succeeds when run individually, but not always when run together. For instance: alphabetical seems to be default on windows but not on linux, but it is not always obvious that this could cause a test to fail on one system and not the other. – Superole Mar 13 '15 at 15:14
  • 1
    Just note you can define this also from command line: `mvn test -Dsurefire.runOrder=reversealphabetical` - this is very useful right for testing of the test independency - to try different order ;) – Honza Mar 06 '18 at 13:20
8

You could create a test suite which runs all of your tests and run that.

With junit 4: -

@RunWith(Suite.class)
@Suite.SuiteClasses({Test1.class,
                     Test2.class,
                     Test3.class,
                     Test4.class,
                     Test5.class
})
public class TestSuite
{
}

That will run them in the correct order.

Peter Jamieson
  • 745
  • 5
  • 10
  • 1
    @Suite.SuiteClasses( { AccountControllerTest.class, BrandsControllerTest.class, UserProfileControllerTest.class, RequestedBrandControllerTest.class, SettingsControllerTest.class } ) When I tried this SettingsControllerTest.class was running before AccountControllerTest.class – ruveena Jul 19 '16 at 10:51
5

If you really need an order of your tests than you should use testng instead of JUnit where you can define dependencies between tests and based on that a particular order of tests. I know that in practice their are times where the independent paradigm does not work.

acabra85
  • 369
  • 4
  • 13
khmarbaise
  • 92,914
  • 28
  • 189
  • 235
3

There is a Maven Surefire plugin that allows you to specify test order.

On the off chance that your tests need to be run in order because they depend on each other, I would strongly recommend against that. Each test should be independent and able to be run by itself. And if each test is independent then it doesn't matter what order they run in. Having independent tests also means you can run a single test repeatedly without having to rerun the entire test chain. This is a huge time savings.

Green
  • 576
  • 7
  • 24
  • 1
    While not an explicit answer as to how to do it in Maven, I did recently find this link in JUnit's docs. https://github.com/junit-team/junit/wiki/Test-execution-order – Green Jan 15 '15 at 14:46
  • Most of what you say is true. Problem arises when you have some expensive tests like UI tests and management does not want to invest in independence of tests as they want short term gains instead of hypothetical long term gains. – Muhammad Hamza Yousuf Feb 02 '21 at 09:30
3

The runOrder parameter of the surefire-plugin will help, but it will only help if the classes are executed in order. It does not help to order the test methods within one class (surefire-plugin 2.22.2, junit 5.6.1). To achieve the order within a test class with jUnit use jUnit's feature of controlling the test order within a class by @TestMethodOrder(MethodOrderer.Alphanumeric.class) (jUnit 5).

John Smith
  • 1,027
  • 15
  • 31
toaster
  • 61
  • 7
2

If your unit tests need to be ran in a specific order, it's probably because your tests are badly designed, or your app is badly designed. Your unit tests should be independant from each other.

Pacane
  • 20,273
  • 18
  • 60
  • 97
  • 1
    This statement is so false I don't know where to begin my answer. Lets say I'm testing a trading system. Lets say my system has 3 major component 1) analyze market, 2) open order 3) report pnl. My unit tests are independent, as they test separate components: 1) build quant model 2) order type decision/execution, 3) analytics reports. To test the workflow, I need to run them in order. I can't report pnl, if I don't have any orders. And I can't open order, if my quant model isn't there. – niken Aug 24 '12 at 20:18
  • 2
    @Nik If you need other entities to test your methods, use Mocks instead of ordering your tests. – Pacane Aug 24 '12 at 21:09
  • 4
    @Nik And after thinking about it, I don't even think (personal opinion here) that testing the workflow has its place in unit testing. I'd do that in integration tests. – Pacane Aug 25 '12 at 14:47