How to stop test if one test fails using vstest.console.exe?

Question

I have a batch file that contains multiple tests defined in a similar form below.

vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t1,t2,t3,t4,t5"

The tests run in order from t1 to t5. However, I want to stop vstest if any one of the tests fails. Is this possible using vstest.console.exe?

Btw, the contents of my test.runsettings is

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <MSTest>
    <ForcedLegacyMode>true</ForcedLegacyMode> 
      <KeepExecutorAliveAfterLegacyRun>true</KeepExecutorAliveAfterLegacyRun> 
  </MSTest>
</RunSettings>

I have checked the Documentation for runsettings, it seems there is no flag/attribute for this case.

I asked about this over on msdn for TFS build servers. No go. Don't know about vstest, tho. — , Apr 06 '16 at 11:37
You reminded me of MSDN. I posted the same question on msdn hopefully they'll have answers then I can post here whatever the official take on the matter is. I think a flag that implements this should be available. But it's nowhere in the documentation. — Bangonkali, Apr 06 '16 at 11:57

score 2 · Answer 1 · answered Apr 06 '16 at 12:46

2

If the number of test to run is small, like in your example, you can split it to multiple runs of vstest.console.exe and check ERRORLEVEL in the batch. If you ERRORLEVEL is not 0, it means a test fails, you can exit the batch.

vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t1"
IF ERRORLEVEL 1 GOTO exit
vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t2"
IF ERRORLEVEL 1 GOTO exit
...

:exit

answered Apr 06 '16 at 12:46

Sergey L

1,402
1
9
11

This is a great suggestion! I will try @lukbl suggestion if it works then I will try this one. I'm planning to implement both to see which is faster. Also implementing the ErrorLevel on a per DLL or test suite basis would be a great idea. – Bangonkali Apr 06 '16 at 12:56

score 2 · Accepted Answer · answered Apr 06 '16 at 12:49

If it is an option for you, then you could introduce base class for tests with cleanup, initialize methods and TestContext property.

In cleanup method you will check if test is failed and by triggering Assert.Fail in TestInitialize you don't allow any other test to pass after that.

[TestClass]
public class BaseTest
{
    private static bool _failAllTests;
    public TestContext TestContext { get; set; }

    [TestInitialize]
    public void InitializeMethod()
    {
        if (_failAllTests)
        {
            Assert.Fail("Fail all tests");
        }
    }

    [TestCleanup]
    public void CleanUpMethod()
    {
        if (TestContext.CurrentTestOutcome == UnitTestOutcome.Failed)
        {
            _failAllTests = true;
        }
    }
}
[TestClass]
public class UnitTest1 : BaseTest
{
    [TestMethod]
    public void TestMethod1()
    {
        Assert.Fail("TestMethod1 failed!");
    }
    [TestMethod]
    public void TestMethod2()
    {
        Assert.IsTrue(true, "TestMethod2 passed!");
    }
}

AlbusMPiroglu · Answer 3 · 2017-02-14T05:10:09.693

In addition to lukbl's answer, you can do the same assembly-wide, so if you have multiple test classes, you'll have a global management on the tests, during vstest.console.exe's run-time (if you're calling it multiple times for example).

Care should be taken depending on how you're employing vstest.console (or mstest). If you're load-balancing between multiple test agents, each test agent will run their own vstest.console.exe, and thus will have their own assembly-level values, so the session management will be limited by the group of tests running on the same agent. Let's say this approach will give you management on the whole set of tests you run with a command: vstest.console.exe /filter: tests.dll

That means regardless of the scope of your session_failed variable (class-wide or assembly-wide) if you end up running different tests from same class with different vstest.console.exe calls, you'll lose the variable value, or the control.

That being said, a simple approach for multi-class test scenario:

[TestClass]
public static class TestSettings
{
    public static bool SessionTestsFailed = false;

    [AssemblyInitialize]
    public static void runsBeforeAnyTest(TestContext t)
    {
        TestSettings.SessionTestsFailed = false;
    }
}

[TestClass]
public class Tests1
{
    public TestContext TestContext { get; set; }

    [TestInitialize()]
    public void MyTestInitialize()
    {
        if (TestSettings.SessionTestsFailed)
            Assert.Fail("Session failed, test aborted");
    }

    [TestCleanup]
    public void MyTestFinalize()
    {
        if (TestContext.CurrentTestOutcome != UnitTestOutcome.Passed)
            TestSettings.SessionTestsFailed = true;
    }

    [TestMethod]
    public void test11()
    {
        Console.WriteLine("test11 ran");
        Assert.Fail("fail the test");
    }

    [TestMethod]
    public void test12()
    {
        Console.WriteLine("test12 ran");
        Assert.Fail("fail the test");
    }
}

[TestClass]
public class Tests2
{
    public TestContext TestContext { get; set; }

    [TestInitialize()]
    public void MyTestInitialize()
    {
        if (TestSettings.SessionTestsFailed)
            Assert.Fail("Session failed, test aborted");
    }

    [TestCleanup]
    public void MyTestFinalize()
    {
        if (TestContext.CurrentTestOutcome != UnitTestOutcome.Passed)
            TestSettings.SessionTestsFailed = true;
    }

    [TestMethod]
    public void test21()
    {
        Console.WriteLine("test21 ran");
        Assert.Fail("fail the test");
    }

    [TestMethod]
    public void test22()
    {
        Console.WriteLine("test22 ran");
        Assert.Fail("fail the test");
    }

And here's an easy way to update all your test initialize methods at once, if their signature is the same, using regexp matching, visual studio replace all: find:

(\s*)public void MyTestInitialize\(\)(\s*)(\r*\n)(\s*){(\r*\n)

replace:

$1public void MyTestInitialize()$3$4{$1\tif (TestSettings.SessionTestsFailed) Assert.Fail("Session failed, test aborted");

and similar for TestFinalize().

How to stop test if one test fails using vstest.console.exe?

3 Answers3