0

I have an Azure DevOps release pipeline setup to run my tests according to this setup:

https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops

My class file contains a unit test with 4 test cases in the following fashion:

[TestCase("test1", "arg1", "arg2")]
[TestCase("test2", "arg1", "arg2")]
[TestCase("test3", "arg1", "arg2")]
[TestCase("test4", "arg1", "arg2")]
public void Test(string name, string arg1, string arg2)
{
    TestStuff();
}

When I run the tests a new release gets created and tests are run successfully however only 3 of the 4 tests are found. test2 is not found for some reason and will just get skipped. From the logs I can see that 146 tests were found in total which is accurate yet only 3 are "discovered"

2020-08-19T23:41:28.3146168Z Number of testcases discovered : 146
2020-08-19T23:41:28.3166614Z ##[debug]PERF: DiscoverTests:AddTestCases: took 0.005 ms
2020-08-19T23:41:28.3290518Z Discovered tests 146 from sources
2020-08-19T23:41:28.3291189Z 
=================================================================
2020-08-19T23:41:28.3291959Z ##[debug]DiscoverTests.PerformDiscovery : Test Discovery completed.
2020-08-19T23:41:28.3295732Z ##[debug]PERF WARNING: DiscoverTests.PerformDiscovery: took 11963.8108 ms
2020-08-19T23:41:28.3607538Z [RunStatistics]This execution slice with id '3', received '4' testcases to execute out of which '3' is discovered.

What does it mean that only 3 were discovered?

If I attempt to run that test by itself the whole thing fails with

Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.

I've verified that the tests are properly associated to a test case via VS (2019) and the other 3 tests run fine it's just 1 test that doesn't get found for some reason.

so cal cheesehead
  • 2,515
  • 4
  • 29
  • 50
  • The error message is cannot found the test assemblies, you should check the .dll file path and ensure it exists in the directory. In addition, I found some similar issue, please also check it. [Issue1](https://github.com/microsoft/azure-pipelines-tasks/issues/7708) and [Issue2](https://developercommunity.visualstudio.com/content/problem/941423/vstest-testrun-task-in-the-release-pipeline-fails.html) – Vito Liu Aug 21 '20 at 13:11
  • @VitoLiu-MSFT The error `assembly not found` only comes up if i try to run a single test with a test case that i know is not "discovered" so i don't think that's the issue. My real problem is that the assembly IS found but only 3/4 tests in a test suite are discovered even though they're all associated. I created a different test suite with 15 test cases and only 13/15 test cases were discovered in that run. – so cal cheesehead Aug 21 '20 at 13:57
  • Please check the field [Test filter criteria](https://learn.microsoft.com/en-us/previous-versions/jj155796(v=vs.140)?redirectedfrom=MSDN) and ensure it is same as local VS 2019 setting. If I configure the filter It will run tests that match the given expression instead of all test case. Please check it and kindly share the result here. – Vito Liu Aug 25 '20 at 10:02
  • @VitoLiu-MSFT I resolved this issue but I'm not 100% what was happening. The tests cases that were not being "discovered", in ADO I removed the associated test case, then in VS2019 re-associated the test case again and that seems to have resolved the issue. Not sure if there's any known issues where the linkage between a unit test and a test case is not working or not but that appears to be the issue. I was not able to reproduce it again. – so cal cheesehead Aug 25 '20 at 17:58
  • Hi, thanks for your sharing, you could accept your answer. In this case, others could directly find the useful solution. – Vito Liu Aug 26 '20 at 08:35

1 Answers1

0

The issue turned out to be a linkage problem between the unit test and the test case. I verified that the existing association was correct but I ended up:

  1. In Azure DevOps test case details removed the Associated Automation
  2. In VS2019 I re-associated the unit test to the test case

I ran the test plan again and that time all test cases were found.

so cal cheesehead
  • 2,515
  • 4
  • 29
  • 50