Ordering Coded UI tests and Results in Test Manager

Question

I have a series of CodedUI test methods that make up a single test case. The test methods need to run in order (IE testmethoda runs then testmethodb and then testmethodc) and I want the results to show up in Microsoft Test Manager to look like testmethoda passed, testmethodb passed, testmethodc failed. Is there a way to do this while being able to run multiple iterations of the overall test case?

I have tried are putting the test methods into a single test method and calling that. This gives me the desired test order and the ability to make multiple test runs, but test manager shows a single pass/fail on the entire test case.

I have also tried attaching a datasource to the individual test methods and ordering them in test manager which gives me the desired test results in test manager but has the side effect that if i want to run more than one data row the order gets messed up. For example 3 data rows would run:

testmethoda
testmethoda
testmethoda

testmethodb
testmethodb
testmethodb

testmethodc
testmethodc
testmethodc

I want them to run:
testmethoda
testmethodb
testmeothdc

testmethoda
testmethodb
testmethodc etc..

I have thought about using an ordered test as well but that still shows up as a single test in MTM and there isn't a method I am aware of to data drive it anyways so it would have it's own problems.

Is there a feature that I am missing in VS or MTM to get these results? Maybe a method that would allow me to define a test run in the results file? Would writing/editing the trx file get my results into MTM? I have a feeling I would also have to make changes to the TFS database which isn't an option.

chaliasos · Accepted Answer · 2012-02-03T10:41:09.077

I don't think there is a way to do this through VS or MTM. The option to add all your test methods on a single one sounds good but when one of them fails then the 'parent' test method stops and throws the 'AssertFailedException' that one of the inner tests had thrown.

However, if your test methods (a, b, c...) are completely unaffected one by the other (this means that if the testMethodA fails the other tests can run without problems), I would try to catch all the internal exceptions and at the end print which methods passed and which not.

[TestClass]
public class TestClass
{
    Dictionary<string, string> testMethods;
    bool testResult;

    [TestInitialize]
    public void TestInitialize()
    {
        testMethods = new Dictionary<string, string>();
        testResult = true;
    }

    [TestMethod]
    public void TestMethod()
    {
        //Run TestMethodA
        try
        {
            TestMethodA();
            testMethods.Add("TestMethodA", "Passed");
        }
        catch (AssertFailedException exception) //Edit: better catch a generic Exception because CodedUI tests often fail when searcing for UI Controls 
        {
            testResult = false;
            testMethods.Add("TestMethodA", "Failed: " + exception.Message);
        }

        //Run TestMethodB
        try
        {
            TestMethodB();
            testMethods.Add("TestMethodB", "Passed");
        }
        catch (AssertFailedException exception)
        {
            testResult = false;
            testMethods.Add("TestMethodB", "Failed: " + exception.Message);
        }
    }

    [TestCleanup]
    public void TestCleanup()
    {
        foreach (KeyValuePair<string, string> testMethod in testMethods)
        {
            //Print the result for each test method
            TestContext.WriteLine(testMethod.Key.ToString() + " --> " + testMethod.Value.ToString());
        }

        //Assert if the parent test was passed or not.
        Assert.IsTrue(testResult, "One or more inner tests were failed.");
    }
}

You could also create a different class that will manage all this behaviour to avoid all these 'try-catch'.

Yeah unfortunately A B and C do affect each other. B and C can't run if A fails so they would have to exit inconclusive. The real problem is that A B and C should all be considered one test case. — stoj, Feb 04 '12 at 06:32
Although your test methods are one test case is a very good practice to devide them in smaller methods so you can later reuse them in other test cases. If for example, the A method is 'LoginToSite()' I believe that more than one test cases use it, so on any change in your Login page you only have to change (or re-record) one method. As for my answer, I have use it on a 'black-box' testing, where a simple call to a Web Service does many database assertions and expects many calls to other applications, so if any of the database assertions or expected calls fails the test will not stop. — chaliasos, Feb 04 '12 at 10:16
Yeah from a development perspective they should be separated, otherwise they would be unmaintable 500 line functions not mention violating DRY....From a test case reporting perspective though I think they should be a single case. — stoj, Feb 09 '12 at 16:40

Ordering Coded UI tests and Results in Test Manager

1 Answers1

Linked