17

We use Spec trait for our tests in ScalaTest. when we run the entire suite, it does not always run in the same order. Most answers in google suggest defining a Suite and specifying all the test names. But this requires us to add the test name every time we add a new test.

Is it possible to use the DiscoverySuite itself and define the test execution order? Like run the tests in alphabetical order. I looked at extending the DiscoverySuite but DiscoverySuite seems to be private to scalatest.

---More info----

By ordering i mean, If there are tests A, B, C.

class A extends Spec  {..}
class B extends Spec  {..}
class C extends Spec  {..}

Then i want the tests to run in order (A, B, C). But what happens now is, it run in a different order everytime.

Udayakumar Rayala
  • 2,264
  • 1
  • 20
  • 17
  • 6
    FWIW, tests should be defined in such a way that execution order does not matter. This makes the individual tests more resilient (changing test A won't break test B), easier to understand (to understand test A, I only have to look at test A) and independent (I can run just the tests I need). – leedm777 Jun 29 '12 at 13:25
  • I understand and completely agree that tests should be defined such that the order should not matter. But Unfortunately these are functional(Webdriver) tests and there are lot in number. So as a first step of fixing them, I wanted to make them run in same order and in the mean time analyse what the issue is. – Udayakumar Rayala Jul 01 '12 at 19:41

2 Answers2

12

DiscoverySuite is private to ScalaTest, yes. The execution order of tests in a Spec (now called FunSpec, by the way) is defined to be the order of appearance in the source file. To define the order of the test classes themselves, you will need to define a nestedSuites method and run that wrapper Suite instead of using Discovery. You can go back to using discovery once you no longer need an order. I'll look at adding a defined order to DiscoverySuite in the next ScalaTest release.

Bill Venners
  • 3,549
  • 20
  • 15
  • As i explained above, i want the tests to run in alphabetical order so that if a test fails, I can reproduce it again. At the moment, we have lot of reasons why a test can fail. Just want to take out the test order out of the equation and analyze other failures. Once the other reasons are less, we will remove the ordering. – Udayakumar Rayala Jul 02 '12 at 11:54
  • 1
    Similar to this, I'm finding that an inheritance model of tests could be beneficial. At the moment, I have test B failing, but prior to its running, test C (that uses B) is being tested, obviously failing. Can I declare that C should be run only if B passes? – akauppi Feb 22 '13 at 11:23
1

See http://doc.scalatest.org/1.0/org/scalatest/SequentialNestedSuiteExecution.html

This seems to provide the specific behavior you're looking for.

F. P. Freely
  • 1,026
  • 14
  • 24