3

I have an n-tier application with the different tiers running on different machines. I need to execute coded integration tests on those different machines, preferably using the MSTest framework since that's what everything else is written in.

There's an orchestration problem, in that before the tests on machine "B" can run, machine "A" needs to be set up correctly. VS out of the box doesn't seem to handle that scenario so I need something else.

On the surface TFS lab manager would seem to be what I need to manage the machines, deploy the code and run the appropriate tests but the wrinkle is that the code doesn't live there - we use Perforce for source control.

So

a) Has anyone successfully used TFS lab manager for test deployment using code that came out of another source control system? If so, how did you set it up?

or

b) Are there any alternative testing frameworks that will let me deploy code to multiple machines, execute coded tests and gather success/failure results?

MarcE
  • 3,586
  • 1
  • 23
  • 27
  • How do you build your code? TFS or something else? – pantelif Jan 10 '12 at 20:14
  • The code I'm testing is built locally using VS. The integration tests are manually kicked off. It could be set up with a TFS build agent if required but given the current infrastructure mix I don't really want to add yet another level of complexity! – MarcE Jan 11 '12 at 08:47

3 Answers3

3

a) I haven't done this with different source control, but I think it's definitely possible with TFS Lab Manager. The TFS Workflow & Test Controller/Agent model works based on Windows Workflow in Lab manager, so you would want to modify the workflow for the build (using the XAML editor or Workflow UI in VS) and replace any TFS source control calls to your own task set to fetch/version/checkin-out code. In fact the TFS Lab workflow is divorced from the build workflow and can just pick up built binaries from a UNC path or something. If you chose the latter, you could build the code with MSBuild and some other driver like Jenkins and then use TFS Lab Manager to execute the tests on each tier correctly.

b) Conversely, you could use Jenkins entirely and the VMWare plug-ins to spin up and down VMs for testing. You could use the Powershell plug-in and powershell remoting or some other MSTest plugin to execute the remote tests. This option seems harder to me, but I think both could work.

Nick Nieslanik
  • 4,388
  • 23
  • 21
1

Based on your comments

The code I'm testing is built locally using VS. The integration tests are manually kicked off.

I would expect any CI platform to operate with a labelled build as input.
Actually on your (both) questions: I think that until you invest the effort to upgrade to a stable Build environment you are doomed to do this by hand/by self-brewed scripts.

Having a build server is a really good thing and, if you are into setting up a modern development environment, you basically can't do without it.

If you 're attracted by TFS Lab Manager, going with TFS-Build is a must. If you go down this path, you should consider toggling everything into TFS - including source control.

I 'd say that using Perforce as source control & TFS for Build/Test is absolutely possible, yet it can be a headache to set up the build properly (literally everything that comes out of the box regarding source control needs to be refactored to p4-friendly calls).

Community
  • 1
  • 1
pantelif
  • 8,524
  • 2
  • 33
  • 48
  • Yeah, that's the conclusion I'm coming to. I like the promise of lab manager (grouping machines as an environment, deploying code, running tests etc) but it really, *really* wants everything in TFS! – MarcE Jan 11 '12 at 12:13
  • What it explicitly demands is TFS Builds. Those are by far easier to setup with TFS source control, but it should be possible to do it with any other source control as well. Another very important factor is your Issue Tracking: I don't know to what extend you could use with Lab Manager any other non-MS tracker. In all, you might be right: Deploying TFS just for the Lab Manager could be a bad choice. – pantelif Jan 11 '12 at 12:28
  • There's an open source plugin for Perforce and MSBuild (see this question: http://stackoverflow.com/questions/2418712/perforce-tasks-for-ms-build). Not sure it'll help but thought I'd mention it. – randy-wandisco Jan 11 '12 at 15:49
0

I'm going to answer my own question with what I actually did: abandoned the lab idea as being just too much work for the job at hand!

It looked promising at first and I'm sure it is possible with sufficient effort but as the other answers noted it would mean substantial edits to the lab workflow to achieve my goal - way more than I can justify. Not only was there a problem with the code not being built with TFS but I also ran into problems trying to tell lab/test manager how to run which tests where.

In the end I cobbled a halfway house solution together using a powershell script to set up the environment (revert hyper-v snapshots, copy a few files around and run some remote installs) and the standard MSTest framework built into VS for executing tests against the environment.

MarcE
  • 3,586
  • 1
  • 23
  • 27