Is there a tool to keep track of manual steps in a CI or CD process?

Question

I'm looking for a tool of some kind that i can integrate into our CI process to keep track of the manual steps we have.

As an example, we want to run through some manual test scripts on the integration server before pushing the version to the test server. Currently QA gets a notification when the build is done, executes the manual testing and then tells someone to push the version to test if it's okay.

What i would love to find is something that will keep track of when the manual tests have been successfully completed and automatically push the version to test.

It should will be possible to notify/trigger the tool from Visual Studio Online and have it trigger the next step in VSO as well.

I've been googling various different things, but can't seem to find anything close to what i'm looking for. Todo list tools like Asana doesn't seem to have the integration point we need, but maybe i'm just missing something?

Can't you automate running these tasks as well? Why is it required to have someone manually start the scripts? — Viktor Benei, Dec 03 '15 at 08:56
sorry, they're not scripts as in code but manual test scenarios. The main issue right now is uploading a file via the UI, waiting for it to be processed and see if the result is correct. The first step is causing us issues, and the last verification may always need to be done by a human — Dennis Newel, Dec 03 '15 at 16:57
"The main issue right now is uploading a file via the UI, waiting for it to be processed and see if the result is correct." -> Is there a reason why you can't upload it via an API? — Viktor Benei, Dec 03 '15 at 17:45
No, but there's logic in the frontend (single page app) that we want to check :) We have unit & integration tests that deals with most of the API, but very few tests for the frontend, which is why we need to run through a set of manual tests before putting it on the test environment. with enough time we could probably automate the whole thing, but there's not enough resources for that right now — Dennis Newel, Dec 03 '15 at 23:53

score 1 · Answer 1 · answered Dec 05 '15 at 07:57

You can use the new Release Management tools in conjunction with test cases to get what you want.

In VSTS you can create test cases to reflect the steps of the tests that you want and then create a Test Plan or Suit to reflect the list of tests that you need to run manually.

Then as part of your release process you could create a custom task that waits for a Test Run to be completed against you list. If that test run has all pass then move to the next step, if any tests fail then fail the release.

This should be fairly easy to setup and you need to call the API to check the Test Run. If your Testers use Microsoft Test Manager you can also have the results associated with the Build that you are deploying and get full traceability.

thanks @MrHinsh. I think it relies on all our tests being in VSTS? we've had a look but can't seem to find out how to hook this up. All our test suites are in TestRails and stories/bugs are in Jira, so it would probably be a larger effort to move everything to VSO just to be able avoid a manual process :) — Dennis Newel, Dec 09 '15 at 00:54

score 0 · Answer 2 · edited Dec 06 '15 at 11:06

0

You can try with the Release Management tool in VSTS. It can achieve partial features you want.

I assume you have three steps in your build definition:

Build solution
Publish to integration server
Publish to test server

You can keep the first two steps in build definition. And then create a release definition in Release Management and add the third steps in it. Configure the definition to "Continuous deployment" and link it to your build definition. Assign your QA to be the approver for this release task. Now, a release task will be created as soon as a build task completes. But it will be pending for the approver (Your QA) to approve. Approve the release task after the test is passed, the build will be published to the test server. Otherwise, reject it.

edited Dec 06 '15 at 11:06

MrHinsh - Martin Hinshelwood

23,409
5
46
69

answered Dec 03 '15 at 09:30

Eddie Chen - MSFT

29,708
2
46
60

This sounds like it could work. One thing that might be missing is actually tracking the steps the QAs do. We currently use TestRails, so maybe there's a way from testrails to trigger an approval in VSO? – Dennis Newel Dec 03 '15 at 19:31
No, there isn't any way to do this for now. – Eddie Chen - MSFT Dec 04 '15 at 05:16
1

Eddie, you can do this with VSTS! – MrHinsh - Martin Hinshelwood Dec 05 '15 at 07:58
@MrHinsh Thanks for your remind:). I mean I cannot find a way to trigger the approval from TestRails. – Eddie Chen - MSFT Dec 07 '15 at 01:01

Is there a tool to keep track of manual steps in a CI or CD process?

2 Answers2