0

Scenario

  • Visual Studio Online build definition.
  • Using a Visual Studio Test - Test Assemblies step to run tests in a specific dll.
  • The tests in this dll are more like build steps (highly customized builders), than tests.
  • The test step has "Continue on error" set to false as I need it to fail when one of the builders fails. And this is working as expected. When a test step fails, the build fails.

Due to changes in test runners, currently no tests are discovered because of a mismatch between the test framework in the dll and the test run adapter used in the build. (I know how to fix this, that isn't the problem.)

The problem is that this only produces an information message in the logs about no tests having been discovered and a warning that there were no testresults to publish, making the build present (on the surface) as successful. A false-positive that I'd like to get rid off.

What do I need to do to make "no tests discovered" fail the build?

I have looked into vsts testrunner configuration with a .runsettings file (https://msdn.microsoft.com/en-us/library/jj635153.aspx), but don't see any options to get this behavior?

Marjan Venema
  • 19,136
  • 6
  • 65
  • 79

2 Answers2

1

The Visual Studio Test task just runs the tests and will fail the build if any tests fail. If you want your build to fail due to lack of tests, there is a task Microsoft Premier Services created called Build Quality Checks (which is free at the time of me writing this). This task allows you to enforce a variety of rules when your build runs. If you set your minimum code coverage to say 10%, it will fail the build if no tests are found.

Additionally, you can right your own task or script to achieve the same goal.

tj-cappelletti
  • 1,774
  • 12
  • 19
1

You can add a Powershell task after Visual Studio Test task. In the powershell task:

  • Get the previous build tasks' detail information by Timeline REST API.
  • Then search the Visual Studio Test task by name such as VsTest - testAssemblies.
  • Check if the parameter for message value is no tests discovered (or No results found to publish etc) issues part. If there has such value, add the command exit 1 in powershell task to fail the build.

The example output for Visual Studio Test task show the message No results found to publish as below:

{
  "id": "29b3a87c-3c5b-473f-9ab5-85eff906250e",
  "parentId": "299295b6-d397-492c-958e-094ec90630f8",
  "type": "Task",
  "name": "VsTest - testAssemblies",
  "startTime": "2017-08-08T07:37:16.18Z",
  "finishTime": "2017-08-08T07:37:23Z",
  "currentOperation": null,
  "percentComplete": null,
  "state": "completed",
  "result": "succeeded",
  "resultCode": null,
  "changeId": 13,
  "lastModified": "0001-01-01T00:00:00",
  "workerName": "name",
  "order": 6,
  "details": null,
  "errorCount": 0,
  "warningCount": 2,
  "url": null,
  "log": null,
  "task": {
    "id": "ef087383-ee5e-42c7-9a53-ab56c98420f9",
    "name": "VSTest",
    "version": "2.0.70"
  },
  "issues": [
    {
      "type": "warning",
      "category": "General",
      "message": "",
      "data": {
        "type": "warning",
        "code": "002003"
      }
    },
    {
      "type": "warning",
      "category": "General",
      "message": "No results found to publish.",
      "data": {
        "type": "warning"
      }
    }
  ]
}
Marina Liu
  • 36,876
  • 5
  • 61
  • 74