1

I am encountering a very strange problem in .net regression testing. I have a test method which fails when I run the complete test suit, but the same test method passes when run individually.

What could be the possible reason behind it. I double checked that other test methods are having no effect on it. The test method uses web service to fetch some data, I do not have any idea why it gets incorrect data when run in the suite.

Has anyone faced the similar situation earlier.

Vertexwahn
  • 7,709
  • 6
  • 64
  • 90
Madhur Maurya
  • 1,016
  • 4
  • 19
  • 42
  • 1
    With what message does the test fail? Have you tried debugging the tests? – Gediminas Masaitis Mar 04 '16 at 10:20
  • 3
    The most possible reason here is that some of your tests that are run before this specific test have side effects, that affect your failing test somehow. – Red Mar 04 '16 at 10:22
  • @raderick How I can ensure that this do not happem. Any ideas or suggestions :) – Madhur Maurya Mar 04 '16 at 10:28
  • @GediminasMasaitis It fails in assert statement. The data received from service is not same as expected Data. I am looking for some guidance if someone has faced similar problem. – Madhur Maurya Mar 04 '16 at 10:33
  • You can debug your whole test suite so that you could see what place throws exception and what setups are provided. Another way is Run test suite disabling groups of tests to see what specific group affects your test. At some point you will find what test is the reason of the problem. – Red Mar 04 '16 at 10:38

3 Answers3

2

I am encountering a very strange problem in .net regression testing. I have a test method which fails when I run the complete test suit, but the same test method passes when run individually.

I've encountered similar things before, usually having to do with:

  1. Application settings. This might be the case with a web service. Note that your tests run in a test application, not in your normal 'exe' application.
  2. Static variables / initialization. If you have a [testmethod] and a [testclass], the class will be instantiated once, after which all methods will be called. If you use state in your class, things can get broken.

In all cases, easiest way to figure out what's going on is probably to put a breakpoint before calling the service.

atlaste
  • 30,418
  • 3
  • 57
  • 87
1

You most likely have test pollution somehow. Check any global state you create to make sure it's reset to the desired state before running a test.

Use copious amounts of logging to get to the source of it :)

CubanX
  • 5,176
  • 2
  • 29
  • 44
0

I have also encountered the same problem and it isn't just exclusive to the Microsoft framework. I've seen this in NUnit as well. For me (and most commonly) it is that you are accessing the same item in multiple tests so that the state of it is not as you would expect when run as a group. It's kind of hard to spot at first but if you have in your mind when writing your tests that any test should be able to run at anytime in almost any circumstances you should be good to go.

To help hunt down what the problem is try Asserting things about the state that you assumed was true (things such as Id == 1, or Name == null, or IsConnected == false) if you can get it to fail before you even start that you can be sure it is somethign to do with state.

Robert Snyder
  • 2,399
  • 4
  • 33
  • 65