0

So my problem is the following. (Microsoft Test Manager, Test controller, test agent)

I have a testplan that run tests on several services, some services are the same but different versions.

The services with different version has the same name on their client dll. Lets say person.client.dll

So here comes the problem.

When I run the tests(the whole testplan) the first version test fails but it doesnt fail when I run them seperatly.

So I googled and did some logging on the testagent and discovered this:

Copying remote file: C:\Users\service.tfstestcontr\AppData\Local\VSEQT\QTController\746\Deployment\Hogia.Person.Client.dll to: C:\Users\fredrik.almen\AppData\Local\VSEQT\QTAgent\746\isptest2013.hogia.local\Deployment\Hogia.Person.Client.dll

So here is the problem if I have pinpointed it correctly. It will copy all the files to the same folder "deployment" The person.client.dll early version will come first then the second, overwriting the first one.

That will cause a failure on the Version 1 tests because it goes against the wrong dll.

So my first thought was. The developers need to rename their dlls. But here is the other issue.

Our customer wants an easy way to upgrade to a new version so having the same name on the dll will simplify their upgrade alot.

So it´s not an option.

I build everything perproject, all of them are different solution but in the same builddefinition.

My question

Is there anyway to fix this? maybe so the copy function seperates alla dlls in different folder or any other solution. I want to keep the folder structure I have when they get into the dropfolder

(Folder structure))

applicationServiceTests

PersonserviceTests

PErsonserviceTestsV2
see link instead : http://prntscr.com/54vm0a

This is how the droplocation looks like, So they are all seperated. Then when I run all tests together all the dlls from all these folders get placed in one folder the deployment folder, thats when dlls with the same name gets owerwritten.

The reason why I have everything in one build definition because this is our deploy acceptance test, if these tests dont run successfully then we cant deploy. So to seperate differente versions to different build solutions will only be time consuming. One testplan for the deploy acceptance test. It´s the devops who run the tests so we need to keep it simple for them and not introduce different testplans for different versions.

(so the solution to have different build_definition will probably not work for me)

Is there anyway to configure the testcontroller/testagent to seperate dlls when doing the copy in a test run. That would solve the problem for me?

Nagik
  • 31
  • 4

1 Answers1

0

You can't deploy tests dlls and their dependencies into subfolders the way you want it.
If you run a Test Case automated by a Unit Test (or Web Test or Ordered Test) the Test Agent looks for a dll containing the test. There is no way to set up which deployment sub-folder to use when run a particular Test Case.

The best solution for you I can imagine is to keep your single build definition and your single Test Plan but not run all tests at once.
Just split your tests into several Test Runs: one separate Test Run for each version of the services your are testing.

Alternatively, you can distribute the test among several environments (one environment for each version of the services) but it will be more expensive than multiple-run approach.

Suggestion how to keep starting the test run and collecting result simple:
(see comments bellow describing how the starting of the tests is currently implemented)

  1. Create a test separate Test Suite for each version of the services to test.
  2. Extend your REST service implementation as followed:
    • iterate over the new created Test Suites and queue as much Test Runs as you needed
      These Test Runs will run one after another
    • wait until all Test Runs are done
    • send one single e-mail containing the report for all Test Runs
Elena
  • 1,928
  • 1
  • 15
  • 23
  • I understand your point to keep the run of the tests as simple as possible, that is indeed very important. How exactly do devops start the tests? Are they using MTM? Do they run a CMD script? Or do you have a build definition that can be queued to start the tests? – Elena Nov 11 '14 at 07:37
  • Today they start the tests with a url that they run from a ps-script. I have written a restservice that talks with the tfs api. The restservice takes in collection, teamproject, testplan and suitid. The service start the run of the tests and sends and email when the run is done. If I split the testrun it will generate one mail and one test report for each run. They got 1 url for each deploy acceptance test and we are 9 companies so they start 9 different testruns on each deploy, if all 9 companies have delivered. – Nagik Nov 11 '14 at 11:15
  • I've just added a suggestion how could you deal with it. Since you are starting _Test Runs_ using _TFS_ API you have a lot under your control. I'm pretty sure there is no way to deal with it running the tests in a single _Test Run_. – Elena Nov 11 '14 at 11:54
  • Thanks for your answer. Was thinking of this a bit for my last resort. But one thing comes to mind, what if I have several services, they all have only one version. But some services have different version of third party binaries. Let say nhibernate or Unity. What will happen then? Because it can only have 1 version of the dll in the deployment folder that the testcontroller creates. But maybe thats another question Still think they did a bad decision to make the logic that all dlls is gathered in one folder =) – Nagik Nov 11 '14 at 15:03
  • You are absolutely right, you will get the same problem if the dlls your are deploying depend of different versions of third party binaries. I'm not sure if they made a bad decision. It is pretty easy to setup the way it is now. It would be much harder if you want to support multy-folder deployment... – Elena Nov 11 '14 at 15:44