1

We are trying to run automated tests using Microsoft Test Manager 2015.

We've almost reached our goal. This is what was done so far:

  • We are able to queue the 'build definitions' and successfully run Coded UI tests on a remote server.
  • We've created a Test Environment and it's in a 'ready' state
  • Test plan has build definition in use
  • Test cases have association to the Coded UI tests.

TFS 2015 build definition More info:

  • We don't have a build and copy to drop off folder steps because we don't have Visual Studio installed on the agent server
  • So far, we manually put the .dll that contains coded UI tests to the C:\build folder on the agent server
  • Test project resides in a separate solution from the actual project we are testing. and we want to keep it this way.

When we try to run a separate coded UI test that is associated with a test case using Microsoft Test Manager 2015, we get an error:

The build directory of the test run either does not exist or access permission is required

  • I don't know how to specify the build directory. And we are not really building anything. My idea is just to run the tests from C:\build folder. Side note: Create Build Drop option is missing from our TFS 2015 vNext build definitions
  • I've included Copy and Publish Build Artifacts step to the definiton, but I think it's useless because we are not even building anything. All DLLs are already on the drive
  • I've checked the permissions for the C:\build folder. the account we are suing has full access to it.

Any help will be greatly appreciated

Update:

  • I've addedd Copy and Publish Build Artifacts step to the definition and specified a network share as a Drop folder
  • Controller Service is running under a domain account
  • Drop Folder has full control given to Everyone, including the controller service domain account
  • I've manually copied assemblies with Coded UI test to this drop folder
  • Controller Service log contains the following:

TcmRunner : Build Directory \\bhc-appqa01\TFSShare\Build Definition 2\

TcmRunner: Error starting the test run: The build Directory of the test run either does not exist or access permission is required

So I'm super confused now... The controller says that it can see Build Directory and the next log message says build directory does not exist.

Denis Molodtsov
  • 802
  • 12
  • 27

3 Answers3

2

I feel you have mixed up a lot of things here.

  1. The screenshot you have here is for new Build definition ie the new 2015 way. These new builds are NOT supported in MTM automated testing workflows. Your build definition must be the older xaml-based builds.
  2. For your scenario the automated testing workflows consume a build drop. So the build that you specify has to have a drop location that can be accessed by the test controller. The older xaml builds provided this.

Good news is that given the fact you are on 2015 you can use the new build workflow to run your automated tests from a test plan/suite

allen
  • 4,627
  • 1
  • 22
  • 33
1

Seems this issue is caused by there are no Visual Studio installed on the build agent.

Microsoft Visual Studio is required to be installed on the build server for the following scenarios:

  • To build any CPP test project, you must install Visual Studio Professional or later.
  • To run unit tests, or coded UI tests, you must install Visual Studio Professional or later.

  • To use data and diagnostic data adapters:

    Code coverage: Visual Studio Premium or later.** Test impact: Visual Studio Ultimate. IntelliTrace: Visual Studio Ultimate.

  • To build any modern style apps on a build machine: Visual Studio Ultimate or Visual Studio Express for Windows 8 (Operating system on
    build server must be Windows 8).

  • To compile and run tests for a project with a faked assembly: Visual Studio Ultimate

Please see the source link from MSDN: Run tests in your build process

PS: As long as you have MSDN there is no additional license for Visual Studio required on a Build Agent.

Update

Seems you still need to assign a build to your test plan that contains your Coded UI tests when your running it with MTM. More details, please refer the similar question: Run CodedUI from MTM 2013

Community
  • 1
  • 1
PatrickLu-MSFT
  • 49,478
  • 5
  • 35
  • 62
  • This is really strange. I actually can run Coded UI tests without Visual Studio on the server with Test Agent. When I queue a build that only contains a Coded UI step it successfully opens the browser and goes through all the steps. The only problem is that I can't run these Coded UI tests from Microsoft Test Manager 2015 – Denis Molodtsov Aug 17 '16 at 13:24
  • Seems you need to **assign a build to your test plan** that contains your Coded UI tests when you running it with MTM. Please see the similar question http://stackoverflow.com/questions/24145805/run-codedui-from-mtm-2013 – PatrickLu-MSFT Aug 18 '16 at 09:48
  • I think you might be right, but I am still doubtful. I hope this is does hold true for vNext (TFS 2015). You had to specify build if you used old XAML build definitions. But I'm using vNext where I can specify a separate "copy artifacts" step. In this step, I copy Coded UI DLLs to a network folder so the definition should 'know' (see the logs) where to get the tests from. On top of that, it's actually working when I queue the builds with no problems. The only problem is with MTM 2015.It's somehow blind and does not 'see' my Build Directory. If you are right,I will have to add a 'Build step'... – Denis Molodtsov Aug 18 '16 at 14:14
1

Back in August 2015 Microsoft announced that they are not investing in MTM anymore (see this blog article Lab management evolution)

In FAQ part at then end you will find following statement:

But, you can safely assume that we will not invest on new features in MTM Lab Center, nor in enhancing the XAML build templates. We will also not bring in any of the above features such as integrations with newer platforms, integration with newer versions of SCVMM, or integration with the new Build and Release Management services into MTM Lab Center. Similarly, we will not integrate the current Lab environments from MTM into the new Build and Release Management services. We will probably not ship another major version of MTM Lab Center.

Elena
  • 1,928
  • 1
  • 15
  • 23