1

I am trying to implement parallel testing using VSTest Task as mentioned in the below article. https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops

Brief description of what I am doing:

I have two self-hosted agents installed in the same server. When I run the tests with a single Agent option (either one of them), it is running without any issue. But when I apply multi-agent option either with a) Simple slicing based on the number of tests and agents or b) Slicing based on test assemblies I am getting the below error. ##[error]The slice of type 'Discovery' is 'Aborted' because of the error: System.Exception: No tests were discovered from the specified test sources.

Error message

Thanks in Advance, Udaya Bhaskar.

Udaya Bhaskar
  • 11
  • 2
  • Can you show your YAML for the testing portion? Specifically, how you're uploading and then downloading your assemblies? It's possible it's not finding anything to test because on the agent it forks to your test assemblies don't exist. – T2PS Aug 26 '20 at 06:30
  • Hi @T2PS, Below is both YAMLs, hope it is sufficient. Publish Artifact YAML steps: - task: PublishBuildArtifacts@1 displayName: 'Publish Artifact: Packages' inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)\Packages' ArtifactName: Packages Download Build Artifacts YAML: steps: - task: DownloadBuildArtifacts@0 displayName: 'Download Build Artifacts' inputs: artifactName: Packages. – Udaya Bhaskar Aug 26 '20 at 10:45
  • Hi @T2Ps, Fyi, My Source control is "TFVC". My build pipelines are using the classic editor. Here are YAML screenshots. [Upload Artifact](https://www.screencast.com/t/lTffmqMwj2) , [Download_Build_Artifacts](https://www.screencast.com/t/WkdoIFOe) – Udaya Bhaskar Aug 26 '20 at 12:34
  • Sorry, I neglected to ask one other detail: the YAML for how you're invoking the tests? – T2PS Aug 26 '20 at 15:01
  • Hi @T2PS Here is the [VSTest YAML](https://www.screencast.com/t/oybEXDCil9) YAML script is to long to fit here, Please let me know if sript is must, then I will send in two parts.. Thanks. – Udaya Bhaskar Aug 26 '20 at 16:49

1 Answers1

0

Based on the YAML in the comments, this looks like a paths on disk issue, from the task help reference.

  • The PublishBuildArtifacts task has a default path of $(Build.ArtifactStagingDirectory); you have overridden this to $(Build.ArtifactStagingDirectory)\Packages. If this is set correctly when you look at artefacts for the build you should be able to download your test assemblies and their dependencies which were uploaded from this location.
  • The DownloadBuildArtifacts task has a default downloadPath of $(System.ArtifactsDirectory) which the YAML view indicates you haven't overridden.
  • The VSTest task has a default searchPath of $(System.DefaultWorkingDirectory); the YAML view indicates you have set this to $(Agent.BuildDirectory)\Bin.

The exact behaviour here may depend on how your agents have been set up for their disk paths. $(Agent.BuildDirectory) will usually be one of the numbered subdirectories from the agent's base working path. Interestingly, while DownloadBuildArtifacts' documentation says $(System.ArtifactsDirectory) is its default, this does not appear in the current predefined variables list; if it in fact refers to $(Build.ArtifactStagingDirectory), this defaults to the "numbered_build_subdirectory\a".

As your test search path will expand to "numbered_build_subdirectory\Bin", I expect that (assuming the files are being correctly published) they are being downloaded to a location that sits outside the search path the test task is targeting, which would explain why no tests are being found.

I would suggest modifying the download and search paths for DownloadBuildArtifacts and VSTest to be the same and relative to the base directory, eg: $(Agent.BuildDirectory)\Tests (or whatever is appropriate for your pipeline).

T2PS
  • 372
  • 1
  • 2
  • 12
  • Hi @T2PS, Thanks for detailed the response. One catch here, **with the same below disk path variables, the single-agent scenario is working well**. I tried with both agents separately. $(Build.ArtifactStagingDirectory)\Packages $(System.ArtifactsDirectory) $(Agent.BuildDirectory)\Bin I did one more experiment: manually verified the dynamic disk path ( C:\DynamicsSDK\VSOAgent\_work\5\Bin ) and run vstest.console.exe manually against test DLL. **Surprisingly It worked well**. Question: When we use multi-agent I hope the same vstest.console is responsible for running tst. – Udaya Bhaskar Aug 27 '20 at 13:47
  • Yes, I would expect the code to be the same. The difference in the scenarios is that in single agent you may be leaving the files from your initial build behind in \Bin and it's these that are being picked up by the test task. In the parallel scenario the test files only exist due to artefact download (particularly if the working directory is cleaned before the job starts) so the path they are being downloaded to needs to be the same as where vstest.console.exe will search. Also, when a build is finished (pass or fail), can you download your binaries from the webUI artefacts list? – T2PS Aug 28 '20 at 01:48
  • HI @T2PS, really appreciate your valuable followup on this. I tried changing the variable paths per your suggestion, Unfortunately, there is same error. Infact all these variables are pointing out to the same location only (Agent.BuildDirectory and System.ArtifactsDirectory). Answering your second question, Yes I am able to access download the binaries from artifucats link. Is it possible to have one screen sharing call next week, so that You can guide correctly. Thanks in advance. – Udaya Bhaskar Aug 28 '20 at 18:58