1

I have a solution that uses the standard TFS build definition template. Unit tests are run when building the solution, but don't fail the build.

I also have a separate testing application that runs a suite of automated tests. These tests take too long to be run as part of the main build (since we need access to the built application in order to do manual testing), but I want to kick off this testing application automatically once a build has completed, and I want it to run against the built application. I don't know if it matters, but the testing application is also part of the solution, and references some of the application DLLs.

I am utterly confused as to how best to achieve this. Do I need to create a new build process template? Do I actually want to have this in a separate build?

Dylan Smith
  • 22,069
  • 2
  • 47
  • 62
Gary McGill
  • 26,400
  • 25
  • 118
  • 202

3 Answers3

1

If you're using the 2013 Build Template (TfvcTemplate.12.xaml) then there is a spot to specify a Post-Test PowerShell script to run. Then you just write a powershell script that runs your custom testing app and call it from the build.

enter image description here

Dylan Smith
  • 22,069
  • 2
  • 47
  • 62
  • OK, thanks - sounds hopeful... But would the custom script / EXE need to run to completion before the build was considered 'finished'? That would be a problem... – Gary McGill Nov 19 '14 at 13:41
  • It's up to you. You can write power shell that waits for it or not. – Dylan Smith Nov 19 '14 at 16:27
1

I would not recommend doing this as part of a build and should instead be part of a release pipeline. Your automated integration tests should be run in an environment that is configured to collect data under test rather than to compile code.

Release Management for Visual Studio 2013 provides the ability to push your build output to a server and then execute whatever you want against it.

http://nakedalm.com/execute-tests-release-management-visual-studio-2013/

I configured a CodedIU test in the example above but really any test setup can be used. I also configured a lab (optional) to asses the environment while the tests were executing.

MrHinsh - Martin Hinshelwood
  • 23,409
  • 5
  • 46
  • 69
  • Thanks, sounds very plausible, but... can you explain why Release Manager is more appropriate than hooking something up using just plain TFS? I'm not using Release Manager currently, and I don't have any experience with it, so I'm sceptical that I can justify the learning curve - or introducing another moving part that needs to be maintained. How would this be better than - say - creating a second "build" in TFS and triggering that from the first build? – Gary McGill Nov 20 '14 at 09:32
  • Release Management is plain TFS as it is part of the 2013 product. You are describing trying to use a compilation tool suited to running Unit Tests (TF Build) to try and do a deployment (however easy) and run Integration Tests. The only tests that run as part of TF Build should be ones that do not need an instance of your application. – MrHinsh - Martin Hinshelwood Nov 20 '14 at 10:06
1

There is third option, in between the two. Use the Lab template to run your tests, see Automated build-deploy-test workflows. You end-up in having the 'normal' build that produce the packaged artifacts, then this other build using LabDefaultTemplate.xaml that picks from there and run your UI or integration tests.

IMHO all three suggestion are good, each with its pros and cons.

Giulio Vian
  • 8,248
  • 2
  • 33
  • 41