0

I am getting the error "Validating stage:The selected stage does not have the right version or settings of the Visual Studio Test task to run tests. Learn more" when trying to run tests from a test plan on Azure DevOps.

Learn more links to this article:

We performed quiet a bit of searching, but none of them worked for us. This could be a simple setting that we are overlooking. Your help is greatly appreciated.

Complete steps to reproduce the problem:

  1. Login to your DevOps account
  2. Create a test plan, call it 'TestPlan"
  3. Add a single test to the plan, and note the test case ID (let's say 65)
  4. On your local machine, open VS 2019 using Git as your source control
  5. Create an MS Test project using C# and .net 4.6.2, call it "TestProject"
  6. Add a single test to the project, name it TC65_UnitTest1, and execute the test
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace TestProject
{
    [TestClass]
    public class TC65_UnitTest1
    {
        [TestMethod]
        public void TC65_TestMethod1()
        {
            // Test 1
            Assert.IsTrue(true);
        }
    }
}
  1. From visual studio Test Explorer, associate the test case "TC65_TestMethod" with test case ID 65.
  2. Push your repo to Azure
  3. From Azure, create a build pipeline follwoing the steps in this article "https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops"
  4. My build yml file is shown bellow
  5. Run the build pipeline
  6. Create a release pipeline with 2 tasks Release pipeline
  7. Run the release pipeline
  8. Open the test plan on Azure notice that test case ID 65 passed
  9. Still on the test plan, click on "Execute"
  10. Select "Run with options" Run with option from test plan
  11. On the dialog pop up, select " Select Automated test using release stage
  12. Click "Run"
  13. Expected the test case to show pass. Instead, I get the following error Run with option error
#AzureDevops.yml
trigger: none

pool:
  vmImage: 'windows-latest'
  demands: vstest

steps:
- task: DotNetCoreCLI@2
  displayName: '1. Build Solution'
  inputs:
    command: 'build'
    projects: '**/*.csproj'
- task: VSTest@3
  displayName: '2. Run Tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        !**\*
        **\bin\Debug\net462\TestProject.dll
      searchFolder: '$(System.DefaultWorkingDirectory)'
      codeCoverageEnabled: false  # Disable code coverage
- task: PublishBuildArtifacts@1
  displayName: '3. Publish Build Artifacts'
  inputs:
    PathtoPublish: 'TestProject'

I followed the article:

I was expecting to execute a test case from the test plan on Azure DevOps, but I encountered an error.

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
WindyCity
  • 1

1 Answers1

0

To Run Automated Test Plans using a Test Plan in Azure DevOps You will need a release pipeline that was created from the Run automated tests from Test Manager template to run tests from test plans in Azure Test Plans.

The Template defines a build stage with a Test Task. The Test Task should have setting as defined below, to allow for the stage to be invoked using pipeline parameters.

enter image description here

Edit:

Adding Additional Screenshot of How the release pipeline receives test.runid as pipeline variable when invoked using run with options.

enter image description here

Daredevil
  • 732
  • 5
  • 13
  • Thank you for taking the time to answer the question. following up on your answer: (1) When you say "You will need a release pipeline that was created from the Run automated tests from Test Manager template", are you implying the use of Test Manager? if so, we no longer TM. (2) in step (11) in the original question, selecting "Test run" in the "Select tests using" dropdown results in the following error: "##[error]Input validation failed with exception: Invalid on demand test run id 0. Has to be greater than 0." – WindyCity Aug 06 '23 at 14:08
  • I have replicated your setup. I created the release pipeline using the + Button shown in Test Plan Settings/Configuration Page. I left the Task Settings unchanged. Can you retry by creating a new release pipeline using the + Button? – Daredevil Aug 07 '23 at 08:01
  • Just to clarify my question. Both the build and release pipelines work as expected, with no errors. The only issues I am facing is how to fix the error message "Validating stage: The selected stage does not have the right version or settings of the Visual Studio Test task to run tests." which occurs after trying to run test cases from the test plan when selecting "Run with options" (step 15 above). please refer to step 18 in the original question for the steps that produce the error. If that works for you, please share the yml files for both build and release pipelines. Once again, thank you! – WindyCity Aug 07 '23 at 16:33
  • The Release pipeline is not a yaml based. As I said earlier, it works with the given settings shared earlier in release pipeline validation stage. For more info: When you run the test through run with options, it basically sends a test run id as a pipeline argument (variable). Added screenshot in above answer. so if you have configured the task to run using test run id pipeliene variable as i did, it will work fine. – Daredevil Aug 07 '23 at 18:17
  • If you don't mind, where did the value 1015484 for the test.RunId come from? – WindyCity Aug 07 '23 at 20:02
  • @WindyCity When you Run the Tests using run with options, and then select an automated run. In the subsequent window you are selecting a build and release pipeline and the stage in release pipeline. Azure DevOps when invoking the stage in pipeline, sets the value for test.runId – Daredevil Aug 07 '23 at 20:04