Is it bad practice to remove and combine regression tests to speed up testing?

Question

I'm working on an app that integrates with a 3rd party web service. I currently have separate integration / regression tests that call the web service to do the following:

Modify Policy - Add Vehicle
Modify Policy - Remove Vehicle
Modify Policy - Add Multiple Vehicles
Modify Policy - Add Insured
...

Most of these tests were created as bugs were found & fixed. The 3rd party web service is slooow and I'm trying to speed the testing process up. Because each test calls the web service, combining them into one test that only calls the web service once would make things much faster.

Would combining these tests be bad practice because each test was written for a specific bug? My concern is that a mistake in refactoring could potentially allow a bug to be re-introduced later on.

score 1 · Accepted Answer · answered Apr 14 '11 at 16:36

Yes, combining them would be a bad practice. Think instead about how to mitigate the risk without combining the tests. One approach - probably your best bet - would be to mock out the web service, so that the tests are much faster without jeopardizing their ability to detect a regression. Another would be to split your slow regression tests into their own suite that is run less frequently (but still frequently enough!) than your usual set of tests. Finally, you could combine them - but I would recommend explicitly reintroducing all the original bugs into your code to verify that the combined test still detects them.

Specific, pointed, direct, unit tests are very valuable; it's nice to know exactly what has broken. Combining tests compromises that value.

score 1 · Answer 2 · answered Apr 14 '11 at 16:56

1

I wouldn't recommend combining them, unless you keep the ability to run them separately (maybe keep them separate in your overnight build, and combined in your continuous build).

Try parallelizing them (on separate 'policies'), if your test framework supports it.

answered Apr 14 '11 at 16:56

artbristol

32,010
5
70
103

I hadn't considered parallelizing them. Good idea! – Seth Reno Apr 14 '11 at 16:58

score 0 · Answer 3 · answered Apr 14 '11 at 16:39

I would suggest including them in your nightly build, so that they do run once a day when you are asleep and not watching the clock. And only removing them your developer time tests.

Of course that assumes they are not soooo sloooow that one night is not enough.

Just combining your tests into one big test is likely to make them useless or worse. Thats's not much better than just deleting them.

Is it bad practice to remove and combine regression tests to speed up testing?

3 Answers3