1

I have an ASP.Net MVC project and I thought I could use a tool like MS Test or NUnit to perform regression testing from the controller layer down to the database, however I hit an issue where tests are not designed to run in order (You can use ordered tests in MS Test, but the tests still run concurrently) and the other problem is how to allow the data created from one test accessible to another?

I have looked at Selenium and WatiN but I just wanted to write something that is not dependent on the UI layer which is most likely going to change an increase the amount of work to maintain the tests.

Any suggestions? Is it just the wrong tool for the job? Should I just use Selenium/WatiN?

ptutt
  • 1,338
  • 3
  • 18
  • 35
  • 1
    Testing from the UI layer, while more complex, provides a great deal of return on value. There is a great explanation here: https://support.smartbear.com/articles/testcomplete/regression-testing/ – Chris Apr 21 '15 at 15:43

3 Answers3

0

Update The link below (and I assume the project) is dead. Best option maybe using Selenium and the Page Object Model. See here: http://code.tutsplus.com/articles/maintainable-automated-ui-tests--net-35089

Old Answer The simplest solution I have found is Rob Conery's Qixote:

https://github.com/robconery/Quixote

It works by firing http requests and consuming responses. Simple to set up and use and provides integration testing. This tool also allows a series of tests to be executed in order to create test dependencies.

ptutt
  • 1,338
  • 3
  • 18
  • 35
  • Yep, looks like Rob has abandoned the project. It looks like there are some good tools that combine with Selenium and expose the Page Object Model to create maintainable regression tests. Take a look here http://code.tutsplus.com/articles/maintainable-automated-ui-tests--net-35089 – ptutt Apr 21 '15 at 23:46
0

Tests should always be independent of each other, so that running order doesn't matter. If your tests depend on other tests you are losing control of what you are testing.

Dan
  • 3,229
  • 2
  • 21
  • 34
  • 1
    I understand this is the general consensus for unit testing, but actually, for my purposes, test dependency is a good thing because it means that the tests test each other. For example, I could write test A for function A and test B for function B. Both test A and B may pass if they are independent, but if they are dependent and the data created by test A is not valid for test B then test B would fail, so test B is testing the result of test A. You may say, but then that means test A is not complete, true, but would anyone claim to have bug free tests? If I could why not have bug free code? – ptutt Apr 03 '12 at 12:27
  • In that scenario, I would say you have Test A and Test B, then you have Test C which is Test A + Test B, and Test D which is Test B + Test A. (Using common code, C and D should be straight forward to write). Running tests randomly would certainly test a wide range of how different features interact with each other, but it would produce inconsistent results, unnecessary test cases (e.g. two unrelated features running after each other), and would make tracking down an issue much more difficult – Dan Apr 03 '12 at 13:27
  • Not to mention most of the testing frameworks are designed with the notion that test are independent of each other, so making them dependent takes more work than not. – heavyd Apr 03 '12 at 14:47
  • If I extend that logic, if I have Test A, Test B ... Test N and Test N could be dependent on the outcome of any prior test, then I should have one test that includes all of the preceding tests. This then is the answer to my question, create one test which contains all of the dependent test cases. – ptutt Apr 04 '12 at 12:38
  • 1
    Sorry, I am marking your response down because I believe it only relates to unit testing, where as my question was relating to regression testing. Independent tests for regression makes no sense. It would be like testing all the individual components for a car, but not testing the car starts and drives after it is assembled. – ptutt Apr 11 '12 at 11:40
  • I didn't say anything about testing individual components. You can have independent tests and still test how different parts of the system interact. The problem with your current setup is that (using your car analogy) your tests might try to test that your engine starts before you put the key in – Dan Apr 11 '12 at 13:36
  • ok, so if I understand your correctly, tests should not call other tests? But within one test you can/should 'test' multiple functions to test dependencies? – ptutt Dec 03 '12 at 02:46
  • Regression tests can and should be ran in a certain order if the scenario means you get a better test. "Tests should always be independent of each other" is just false. – Chris Apr 21 '15 at 15:34
0

WatiN, and I'm assuming Selenium, won't solve your ordering problem. I use WatiN and NUnit for UI automation and the running order is not guaranteed, which initially posed similar problems to what you're seeing.

In the vein of what dskh answered, you want independent tests, and I've done this in two ways for Integration / Regression black-ish box testing.

First: In your test setup, have any precondition data values setup so you're at a known "good state". For system regression test automation, I've got a number of database scripts that get called to reset data to a known state; this adds some dependencies so be conscious of the design. Note: In straight unit testing, look at using mock objects to take out dependencies and get your test to be "testing one thing". Mock objects, stubbing method calls, etc is the way to go if you can, which based on your question sounds likely.

Second: For cases where certain things absolutely had to be setup in a certain way, and scripting them to test setup added a ridiculous amount of necessary system internal knowledge (eg: all users setup + all permissions setup + etc etc etc) a small number of "bootstrap" tests were setup, in their own namespace to allow easy running via NUnit, to bootstrap the system. Keeping the number of tests small and making sure the tests were very stable was paramount. Now, on a fresh install, the bootstrap tests are run first and serve as advanced smoke tests; no further tests are run if any of the bootstrap tests fail. It is clunky in some ways, but the alternatives were clunkier or more time/resource/whatever consuming.

OCary
  • 3,231
  • 2
  • 16
  • 17