0

I have a set of mocha tests that launch Spectron to do end-to-end testing on an Electron application.

When I try to run the tests through npm run, they just time out and fail. It seems that the application isn't launching. There is a Run Functional Tests build step template in VSTS, but that's for an actual test DLL. Is there something similar for Javascript end-to-end testing?

Jon
  • 2,644
  • 1
  • 22
  • 31
  • Do you know if the Spectron framework requires you to run on the desktop to run those tests? If you are running on the Hosted Build Pool you are running as a service not in a Desktop session. – Chris Patterson Dec 08 '16 at 20:50
  • 1
    The test needs to interact with window and run on build agent machine, so the build agent need to run in interactive mode, in other words you need to setup a build agent to run in interactive mode. (Hosted agent is running in service mode) Does your build agent run in interactive mode? – starian chen-MSFT Dec 09 '16 at 04:38
  • You need to configure your own build agent and run it in interactive mode as starain mentioned. – Eddie Chen - MSFT Dec 14 '16 at 01:34

2 Answers2

0

The Hosted Agent is running as the service, however launch Spectron to do test with Electron app need to interact with window and the test is running on build agent machine. So you need to setup a on premise build agent to do that test (works fine for me with this test sample. Build steps: npm install, npm install test)

  1. Log on to the machine using the account for which you've prepared permissions as explained above.
  2. Go to the Agent pools control panel tab
  3. Click Download agent.
  4. Click Windows.
  5. Click the Download button.
  6. Run PowerShell as Administrator.
  7. Run the commands under Create the agent.
  8. Run .\config cmd

More information to setup on premise build agent, you can refer to this article.

starian chen-MSFT
  • 33,174
  • 2
  • 29
  • 53
  • Using an on-prem build agent is not the best solution. They should use the Functional Testing task instead. – MrHinsh - Martin Hinshelwood Dec 09 '16 at 07:21
  • @MrHinsh Run Functional Tests task does not apply to Jon's scenario since mocha and spectron run the test base on Node.JS & browser. VS Test agent cannot run this testing. – Eddie Chen - MSFT Dec 14 '16 at 01:33
  • I was able to get the steps going. I have the agent locally and can verify that the build runs on it. However, the tests still don't seem to be running even on my local machine as the build agent. Is there something else I may be missing? – Jon Dec 14 '16 at 12:20
  • Disregard my previous comment...I was running it as a service instead of interactive mode. It now works! Is there any reason it doesn't work as a service, though? Having it as part of a CI would require the agent to always be running interactively. Probably not a huge deal, but just curious. – Jon Dec 14 '16 at 13:32
  • @Jon Because aunch Spectron to do test with Electron app need to interact with window, so agent need to be running interactively. CI build doesn't request build agent running interactively. – starian chen-MSFT Dec 15 '16 at 01:26
  • If you are running tests that need desktop interaction then you should be using Functional Testing and Machine Groups. It is neither a good idea, nor a good practice to do that during a compilation process. – MrHinsh - Martin Hinshelwood Dec 16 '16 at 15:14
  • @MrHinsh yes, but not all of test framework could be run in Functional Testing. – starian chen-MSFT Dec 18 '16 at 05:07
  • @starain-mart, incorrect: ALL test frameworks can be run through functional testing. All you need is a Visual Studio Test Adapter that supports it. If one has not already been built for that framework then it is easy to create your own. See http://mortenhoustonludvigsen.github.io/JsTestAdapter/CreatingATestAdapter/ – MrHinsh - Martin Hinshelwood Dec 22 '16 at 03:08
0

Since the tests need to interact with the desktop they are not a good fit for compile time where you dont usually have a running instance of your application. Stick to pure unit tests at build time. A better fit would be to use Release Management to orchastrate those tests.

That said, you can easily deploy your application (build or release) and the. Use the Functional Testing task to initiate your UI interactive tasks. You need a machine with the Test Agent deployed, but there is a task for that too.

enter image description here

Since Functional (end to end) tests interact with the desktop you can only run one at once. This is fine if you only take a few minutes, but as you get more functional teats this can take way longer. The Functional Testing task allows you to split the test execution across multiple agents so that you can run the tests in parallel.

Since your alternative is to configure your own Build Agent in interactive mode you need a server anyway. You would be more future proofing to use this model from the start as you can scale.

If a Visual Studio Test Adapter is not available for your framework then you can create your own to let Visual Studio, and additionally the Functional Testing plugin, detect and execute any test framework. For Javascript you will find http://mortenhoustonludvigsen.github.io/JsTestAdapter/CreatingATestAdapter/ has some awesome examples of how to create a Jasmin Test Adapter as an example.

MrHinsh - Martin Hinshelwood
  • 23,409
  • 5
  • 46
  • 69