0

I have a Jenkins job that runs Selenium tests on Browserstack via the Nightwatch.js framework. We have an entire suite of tests that nightwatch runs in separate processes, and we need a way to get the pass/fail value back to Jenkins after all tests have run.

I have been trying to use the Nightwatch hook teardown to run a piece of code at the end of each module with if(this.results.failed) { take action }, but I can't figure out what action I need to take in order to make the failure accessible to the Jenkins job.

My initial thought was to create an environment variable, and set it to false any time a test fails, but I think each process runs in a separate subshell, so the variables created/modified by the test module are not accessible.

My second idea was to create a counter in the globals module, but that value does not increment as expected when referenced by a global after method.

imthefrizzlefry
  • 21
  • 8
  • I don't have much experience with nodejs. but if you could exit with status code other than 0, your jenkins job will fail. try that in `{take action}` – Gaurang Shah Oct 06 '17 at 08:19
  • I actually did try this, we are using the gulp-nightwatch module, and it turns out if you exit the test with status != 0, gulp-nightwatch doesn't output any details to the console. It only prints that gulp-nightwatch encountered an error, and our reporter for that test suite fails to run. – imthefrizzlefry Oct 06 '17 at 18:29

2 Answers2

1

I was able to solve this issue using the Jenkins Log Parser Plugin to fail the test in the event that the console output contains the string "TEST FAILURE".

This was accomplished by creating a rule file that contains a single line:

error /TEST FAILURE/

imthefrizzlefry
  • 21
  • 8
0

Here is how I would recommend you do it:

  • Create a synchronised writer in the afterEach function (https://github.com/nightwatchjs/nightwatch-docs/blob/master/guide/using-nightwatch/test-hooks.md) and write the test results to a flat file(csv, tsv, txt etc) on the system running jenkins. You may choose to write only failures if(this.results.failed) OR write all testIDs with pass/fail status to the file
  • Once your testing build has completed, you will have all the details in the flat file (including details on failures).
  • You may now trigger a downstream job (a different job on Jenkins using downstream jenkins plugin) which has a simple script(shell script) looking for failures in the test output file you wrote in step 1. Using this, Jenkins will have details if there were test failures

Note: If you have multiple independent builds running tests, please ensure you are using the jenkins build blocker plugin to start the downstream job after all independent builds are completed

BountyHunter
  • 1,413
  • 21
  • 35