6

According to the official TestCase documentation, you can use a "not contains" (!~) filter in a TestCase filter.

I'm trying to use this filter in an Azure Devops pipeline, but the filter is not supported according to the error I get when running that pipeline:

##[error]Incorrect format for TestCaseFilter Error: Invalid Condition 'FullyQualifiedName!~DSN'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

This functionality was added in a 16.0 preview version. The pipeline is using VSTest 16.6.1, as evidenced by thes lines in the output:

C:\agent\_work\_tool\VsTest\16.6.1\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\agent\_work\_temp\xulnpcmgdnx.tmp"
Microsoft (R) Test Execution Command Line Tool Version 16.6.1

However, the "not contains" operator is still not supported.

I have tried:

  1. Using a preview version of VSTest (16.7)
  2. Using "Visual Studio 2019" as test platform version in my VsTest pipeline task (as opposed to directly using a VSTest version)
  3. Using a VSTest version prior to the release that included the "not contains" test filter, which showed the exact same error as expected

I'm at a loss, is there some other kind of dependency that I'm not aware about? Could this be an issue / bug on Microsoft's part? Any help or pointers would be greatly appreciated.

Joshua VdM
  • 628
  • 5
  • 16
  • Hi friend, not getting your latest information about this issue. Does it still persists to block you, feel free to let me know if there's any update. I'm willing to help :) – LoLance Jul 16 '20 at 09:31

2 Answers2

0

This issue could be related to the VSTest version you use. Here're some tests on my side:

  • Test Execution Command Line Tool Version 15.9.1(Hosted VS2017 agent,Failed):

enter image description here

  • Test Execution Command Line Tool Version 16.6.0(Hosted Windows2019,Succeeded):

enter image description here

  • Test Execution Command Line Tool Version 16.7.0-preview-20200519-01(Self-agent,Succeeded)

enter image description here

enter image description here

So i suggest you can run the pipeline with windows 2019 agent first to check if the issue comes from VSTest versions. Then choose correct version.

LoLance
  • 25,666
  • 1
  • 39
  • 73
  • I'm not sure I understand exactly what you're getting at - I showed that I'm running VSTest 16.6.1. The "not contains" operator was added in a 16.0 preview. – Joshua VdM Jul 07 '20 at 08:14
  • @JoshuaVdM Please check my pics above, it works well for 16.0.0 and 16.7.0-preview-20200519-01 in my machine. It proves that the operator was actually added in 16.0. I'm not sure if there's something wrong with 16.6.1, can you run the test with Micirosoft-hosted agent to test that? The hosted agent has 16.6.0, let's check if the issue is about the 16.6.1 you use. – LoLance Jul 07 '20 at 08:20
  • 1
    I see what you mean now - I'll try with a microsoft-hosted machine with VS2019 to be sure! – Joshua VdM Jul 07 '20 at 08:28
0

Update the MS Test NuGet Packages

I got error, Incorrect format for TestCaseFilter Error: Invalid Condition 'FullyQualifiedName!~Integration'. on my build server.

My code was using a version 1 of MS Test and .NET 7. Update your MS Test nuget packages in your tests project.

Right click the project and select Manage NuGet Packages...

enter image description here

Now my build and test works!

I have experienced this before as well. Older MS Test is missing features like this and [DataRow].

Jess
  • 23,901
  • 21
  • 124
  • 145