2

With continuous integration, it makes a lot of sense to re-run failed unit tests first the next time a cycle starts. This shortens the feedback loop.

Is there any known way to achieve this with CCNet and NUnit?

In principle, CCNet would have to have "memory" of which unit tests failed and NUnit would need to be instructed to run them in a different order.

Pieter Venter
  • 21
  • 1
  • 2
    If your unit tests take that long to run that the order of importance is that critical, may be a sign they need some evaluation. Also, are the developers running the unit tests at the work station prior to check in? That way they can tackle the problem proactively instead of relying on the build server. – Aaron McIver Oct 14 '10 at 14:59
  • 1
    Just to clarify - order is not important for the unit tests to succeed. It could well run in the original order or randomly for that matter. It's been demonstrated by the JUnit community that reordering of tests with a continuous integration cycle has benefits. You want to get feedback as soon as possible. Failed tests or new tests might as well run first to help this cause. – Pieter Venter Oct 14 '10 at 15:49

2 Answers2

3

This would be against a principle of (unit) testing, that says that the ordering of tests has to be totally random (to make sure, that there is no dependency whatsoever between the individual tests).

So the answer is: No.

Btw.: How would this make sense and shorten the feedback cycle?

Thomas

Thomas Weller
  • 11,631
  • 3
  • 26
  • 34
  • There are no dependencies and this is not what it's about. If you are going to run 600 tests it simply makes sense to run the ones that have failed first. – Pieter Venter Oct 14 '10 at 15:45
  • Why? I don't get it. All the tests are run anyway and the test report shows you clearly the failing ones. So why would the order make any difference? – Thomas Weller Oct 14 '10 at 17:38
  • It saves you a bit of time, especially if you have thousands of unit tests that could take minutes to complete. If one fails, it's a convenience that next time you run failed ones first. Often times you want to fix one and verify the test passes now. Just convenience thing really. – Pieter Venter Oct 15 '10 at 17:00
  • How? Normally, you have a CI server, that doesn't even have a display and you get some sort of build notification only when everything is finished... – Thomas Weller Oct 15 '10 at 17:47
  • CI 1.5 has Web UI component that gives basic feedback. You can see build as well as unit test progress. – Pieter Venter Oct 15 '10 at 20:16
  • Oh, didn't know that. It's quite a long time that I worked with CC.Net. In this special case, what you have in mind might really be useful... – Thomas Weller Oct 16 '10 at 02:27
1

An option would be to run the nunit command line with parameters. Check the parameter /run for instance:

/run=UnitTests.TestMainProgram.Test_Sum_NegativeValues_ResultCorrect

You can specify what you want to run with a good granularity, from an entire namespace to a function. If you have your tests organized in such way that the 'top offenders' are grouped by namespace or class you can achieve it. The problem you can find is how to execute 'the rest' of them whithout having to mantain the build configuration everytime you add a new namespace. Depending on your classes setup it is feasible or not.
If you badly need this another option can be to organize your tests in different assemblies.

Cristian T
  • 2,245
  • 3
  • 25
  • 45