How can I add a non-test-method as test during test execution in MSTest?

Question

I have the following setup to run my tests in parallel (for Selenium tests):

[TestClass]
public class ParallelTests
{
    [TestMethod]
    public void TestAllParallel()
    {
        var testActions = Assembly.GetExecutingAssembly().GetTypes()
            .Where(t =>
                t.GetCustomAttribute<CompilerGeneratedAttribute>() == null &&
                t.GetCustomAttribute<TestClassAttribute>() != null)
            .Select(t => new Action(() => RunTest(t)))
            .ToArray();

        System.Threading.Tasks.Parallel.Invoke(testActions);
    }

    private static void RunTest(Type test)
    {
        var instance = Activator.CreateInstance(test);

        ExecuteTestMethods(instance);
    }

    private static void ExecuteTestMethods(object instance)
    {
        var testMethods = instance.GetType().GetMethods()
            .Where(m =>
                m.GetCustomAttribute<TestMethodAttribute>(false) != null &&
                m.GetCustomAttribute<IgnoreAttribute>(false) == null)
            .ToArray();

        foreach (var methodInfo in testMethods)
        {
            methodInfo.Invoke(instance, null);;
        }
    }
}

This works fine although the test result report on my Jenkins server only shows that one test has run. I would like to see in the report all the tests that has been run by TestAllParallel().

Is this possible to do? I'm thinking that perhaps it would be possible to add a method as a test to MSTest during runtime.

I don't think mstest has a method to add tests at runtime. You could create a separate trx file for your tests (generating the file in your methods) and then the mstest plugin in Jenkins would think that it was output from MSTest. — John Koerner, Apr 03 '16 at 13:52
Maybe you could try to check the code-coverage for the test code itself (instead of the code that being tested). Though I'm not sure how to do that in jenkins & mstest. — Thariq Nugrohotomo, Apr 09 '16 at 01:37
why do you execute all your tests on your own and not instructing your Jenkins server to do it for you? — silver, Apr 12 '16 at 07:03

score 1 · Accepted Answer · answered Apr 06 '16 at 13:34

I am afraid it is not possible to add a method, but you could report results from all runs in that specific Test. First wrap execution of test methods with something similar to code below so you can save results of runs:

    private static void ExecuteMethod(MethodInfo method, object instance)
    {
        try
        {
            method.Invoke(instance, null);
            threadSafeStringBuilder.Append("Test: " + method.Name + " passed");
        }
        catch (UnitTestAssertException utException)
        {
            threadSafeStringBuilder.Append("Test: " + method.Name + " assertion failed" + utException.Message);
            _allPassed = false;
        }
        catch (Exception ex)
        {
            threadSafeStringBuilder.Append("Test: " + method.Name + " exception: " + ex.Message");
        }

    }

Later in testAllParallel:

public void TestAllParallel()
{
//...
System.Threading.Tasks.Parallel.Invoke(testActions);
if(_allPassed){
    Assert.IsTrue(true, "All tests Passed:\n"+threadSafeStringBuilder.ToString());
    }else{
    Assert.Fail("Some tests failed:\n"+threadSafeStringBuilder.ToString());
    }

)

Please keep in mind that it is just an idea, not completely working code. In particular there is no ThreadSafeStringBuilder in .net so probably you need to implement your own or use some library.

score 0 · Answer 2 · edited May 23 '17 at 12:07

There is another system that might work just as well, using System.Diagnostics.WriteLine. It will output whatever you write in it. Here is another SO question discussing it: Add custom message to unit test result

A very interesting point in that thread is that without a status message someone could "sabotage" your tests by removing all of a test content so that it passes, so you might need to send diagnostics in normal tests anyway.

score 0 · Answer 3 · answered Apr 12 '16 at 15:31

I'm not sure that I understood correctly your question but I think that you should change your solution a bit in order to overcome this setback. Try to build in Jenkins a project that only execute command line, the command can be to execute all your tests with the MSTest command. you can see how to do it here

How can I add a non-test-method as test during test execution in MSTest?

3 Answers3