2

I have a set of manual test cases which I have run successfully against my development environment. I would now like to run the same tests against the QA and UAT environments (and maybe later against the STAGE and PRODUCTION environments).

My test cases include the URL of the site to test. In my situation, the URL differs from one environment to the next, for instance dev.mysite.com, qa.mysite.com, uat.mysite.com. The test steps will be identical in the QA and UAT environments to the steps which have run successfully in the DEV environment.

I know that I could create separate test plans for QA and UAT, and that I could clone the existing tests into the new test plans. That would allow me to edit the cloned test cases to change the parameters.

But this would be a maintenance problem: for every test, I would have three separate copies. When a test changed in DEV, I would have to make the same change to the QA and UAT versions!

Is there a better way to accomplish running the same tests in multiple environments?

Note that these are existing physical environments, so I can't use Lab Management to create a set of isolated virtual environments, where I could play with networking to have them all called env.mysite.com, or something.

John Saunders
  • 160,644
  • 26
  • 247
  • 397
  • Where do (would) the test parameters come from in this case (the dev.mysite.com)? App.config? Would a config transform help in this case, with a different build configuration for each environment and corresponding build definition in TFS? – Brendan Green Sep 22 '14 at 03:26
  • I'm wondering if there's some way to specify it per test environment or something. It's the test that needs the parameters, not the application being tested. In addition to URL (I said just URL to keep it simple), there may be different username/password per environment, or something else. These aren't program params. If this was unit tests or something based on MSTEST, I could make it data driven. I don't know what to do with manual tests. – John Saunders Sep 22 '14 at 04:31

1 Answers1

0

There is not a way to pass parameters as the primary use case for manual testing / and coded UI is validation of functional behaviour. Once you have validated it once a simple smoke test should be used to validate the deployment.

However you can use the hosts file on the client machines to have a constant url, say m siteundertest.mydomain.com always point to the correct server.

You then are simulating a single url when in fact it points to a different box.

MrHinsh - Martin Hinshelwood
  • 23,409
  • 5
  • 46
  • 69
  • I was referring to manual tests, not coded UI. For coded UI, I presume I can use the MSTEST data binding feature. – John Saunders Sep 22 '14 at 17:13
  • 1
    If its manual why cant the Tester just enter the correct URL? – MrHinsh - Martin Hinshelwood Sep 22 '14 at 17:31
  • Entering the correct URL would work if the only difference between the environments were the URL. Unfortunately, the test data are different between the environments. In particular, different sets of customers exist in each environment. For many tests, it's necessary to query the database to find the customers who meet particular criteria required by the test. For instance, a test might need to use a customer who has purchased more than a certain amount of products from a particular product line within a particular period of time - to test discount logic. – John Saunders Dec 11 '14 at 20:20
  • Of course, the ultimate solution is to get the same set of test data in each test environment. But even so, dynamic changes can happen to the data. In the discount example, maybe a particular discount can only be given once, so that it might be necessary to change which customer is used for the test after the test has run successfully. – John Saunders Dec 11 '14 at 20:22
  • 1
    If the test is parameterized, then I believe that a Coded UI test generated from a manual test run will have parameters data-bound to the test case work item. If I wanted that Coded UI test to work with different environments, then I might change the data source to something file-based, and use different files for different environments. I guess I'm asking for the same feature for manual tests - to be able to switch out sets of parameters without needing to make a copy of the test case, with only the parameters being different. – John Saunders Dec 11 '14 at 20:25
  • It is a recommended approach that tests are not environment specific. You should use Environment variables in the environment. Deployments to different environments can managed by using Release Management. RM will manage your veriabled by environment for you... – MrHinsh - Martin Hinshelwood Dec 13 '14 at 06:10
  • 1
    My test steps are not specific to the environment. Unfortunately, the test _data_ ***is***. – John Saunders Dec 13 '14 at 18:08
  • Use a environment variable to tell your tests what data to load. – MrHinsh - Martin Hinshelwood Dec 14 '14 at 16:48
  • And test data should not be environment specific... That's definitely an anti-pattern. I would look at what this is the case... – MrHinsh - Martin Hinshelwood Dec 14 '14 at 16:49
  • Thanks, can you point to how to use environment variables to tell tests which data to load? – John Saunders Dec 14 '14 at 21:14
  • Not sure what you mean... just create an environemtn veriable called "TestSet" and set it to the apropriate value. Then build your tests such that they support multiple data sets. – MrHinsh - Martin Hinshelwood Dec 15 '14 at 07:39
  • Manual test should be identical across environments. – MrHinsh - Martin Hinshelwood Dec 15 '14 at 18:16
  • I agree that would be ideal. I should also make clear that when I say "environments", I mean: Local Development, Integration, QA, UAT, Production. The data are different across these environments. Unfortunately. I wind up duplicating the tests and changing the parameters to accommodate the different sets of data. – John Saunders Dec 15 '14 at 18:24
  • 1
    Have you considered storing the data somewhere else? Like a database or as files that you can deploy? – MrHinsh - Martin Hinshelwood Dec 16 '14 at 06:26