Pass argument or option to Unit Tests from VSTest.Console.exe

Question

I can successfully run the VS unit tests from the command line (and hence from the build on the build machine).

VSTest.Console.EXE "MyTest.dll" /logger:trx /platform:x64 /inIsolation

I can also filter out any required tests that I don't want to execute on a certain environment with /TestCaseFilter option:

VSTest.Console.EXE "MyTest.dll" /TestCaseFilter:Name!=Verify_DigitallySigned

This is needed to not to run "check if digitally signed" test(s).

This way I can filter out the required set of test case/s.

However, what I want is to let the unit test know if certain tests (asserts) are not required. For example passing a "/DontTestSigning" argument. This way the unit tests (written in C++ or C#) would see such parameter/option, and would not do additional asserts, thus preventing the build failures on not-real production builds (such as on PR builds).

I see that there is /testsettings option with VSTest.Console.exe (and with MSTest.exe also), but I am not sure how (IF) that can be applied and letting the actual test functions to know about some "dont-do" option.

something like `if(executeTheTest) Assert.That(...)`? Having said this there´s no way to provide from the outside of your tests which lines of code should actually execute. That is in no way different from any program you´ve written. The program itself surely needs some switch, e.g. by checking against `executeTheTest`. — MakePeaceGreatAgain, Jun 28 '19 at 12:15
The only approach I've envisioned is environment variable. Looking for better option — Ajay, Jun 28 '19 at 12:31
Managing multiple environmental variables would be too much. I would rather maintain an xml/json file to have key values pairs like (TestCase, parameter to decide whether a specific assert should be skipped) to make it more granular. An helper class winch parses and helps in arriving at the condition should be written and can be used to skip the asserts. — SolidMercury, Jul 02 '19 at 15:47

score 8 · Accepted Answer · answered Jul 02 '19 at 06:43

You can also provide a .runsettings-file to the vstest.console-process, as indicated here. https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019.

In order to provide custom parameters modify the TestRunParameters-section, e.g:

<!-- Parameters used by tests at runtime -->
<TestRunParameters>
  <Parameter name="executeAsserts" value="1,2,3" />
</TestRunParameters>

These parameters can now be accessed via this code:

TestContext.Properties["executeAsserts"];

so that your final test-code may look like this:

[Test]
public void MyTest()
{
    var assertsToRun = TestContext.Properties["executeAsserts"].Split(",").Select(x => Convert.ToInt(x)).ToArray();
    if(assertsToRun.Contains(1)
        Assert.That(...);
    if(assertsToRun.Contains(2)
        Assert.That(...);
    if(assertsToRun.Contains(3)
        Assert.That(...);
}

You should be able to run the test using the following command:

vstest.console.exe MyTestAssembly.dll /Settings:MySettings.runsettings

You can also do this without an extra file, e.g. `vstest.console.exe MyTestAssembly.dll -- 'TestRunParameters.Parameter(name=\"executeAsserts\", value=\"1,2,5\")' — Overlord Zurg, Jan 24 '23 at 23:18
NB if you want to debug those test you may need VSTEST_DEBUG_HOST=1 in the VS Project Properties Debug settings. — osullivj, Jun 26 '23 at 17:03

MakePeaceGreatAgain · Answer 2 · 2019-06-28T12:32:55.020

3

Basically you should favour to have only a single Assert within your test, so that every tests checks for one single thing.

So what you have is similar to this, I suppose:

[Test]
public void MyTest()
{
    Assert.That(...);
    Assert.That(...);
    Assert.That(...);
}

When you want to exclude e.g. the second Assert, you have of course to provide some functionality in your code to execute or not to execute those lines, e.g.:

public void MyTest()
{
    Assert.That(...);
    if(executeSecondAssert)
        Assert.That(...);
    Assert.That(...);
}

You can introduce some compile-switch that sets the value for the above bool-flag:

#if(EXECUTE_ASSERT)
    bool executeSecondAssert = true;
#else
    bool executeSecondAssert = false;

and now provide that compile-switch via an environment-variable.

edited Jun 28 '19 at 12:32

answered Jun 28 '19 at 12:22

MakePeaceGreatAgain

35,491
6
60
111

That would require compiler/preprocessor switches to be customised for different builds. The code won't be common then. – Ajay Jun 28 '19 at 12:30
Even I would prefer single asserts. But that's not always the case. Good test cases need few lines of asserts to be succeeded before actual test succeeds. Performance and time of total build time is one reason, among others. Hence conditional runs of some test cases are required for few tests on my end. – Ajay Jun 28 '19 at 12:33

Pass argument or option to Unit Tests from VSTest.Console.exe

2 Answers2

Linked