TravisCI passes despite tests failing, Gulp setup with tests on Jasmine + Frisby

Question

I'm using the gulp-jasmine-node plugin to run my Jasmine/Frisby.js tests like so:

gulp.task('runJasmine', function() {
  return gulp.src(['test/*spec.js'])
    .pipe(jasmineNode({
       timeout: 10000
    }));
});

gulp.task('test', ['runJasmine']);

Running gulp test locally I get the following (snippet only):

2) Frisby Test: Get API root
[ GET http://localhost:8080/api ]
Message:
 Error: Header 'content-type' not present in HTTP response
Stacktrace:
 Error: Header 'content-type' not present in HTTP response
at null.<anonymous> 
...

Finished in 0.042 seconds
1 test, 2 assertions, 2 failures, 0 skipped

My .travis.yml:

language: node_js
node_js:
  - "0.12"
  - "4.2.4"
  - "5.3.0"
before_script:
  - npm install -g gulp
script: gulp test

And the corresponding raw CI output on Travis (end snippet only):

Finished in 0.054 seconds
[31m1 test, 2 assertions, 2 failures, 0 skipped
[0m travis_time:end:00bb9734:start=1455472901504935117,finish=1455472903105906383,duration=1600971266
[0K [32;1mThe command "gulp test" exited with 0.[0m

Done. Your build exited with 0.

I'm not sure why the build is passing despite failures - is there a certain configuration in Travis when using gulp, depending on test tool used? Is it due to the process exiting with code 0, and if so, do I need to modify my gulp command to exit the process on failure (not ideal as would interrupt other tasks in the runner).

Can't seem to find this problem or identical setup pre-addressed. Please help! Also, this is my first Stack Overflow Q, so hopefully I've provided all info needed. Thanks :)

Do you have a link to the Travis Project? Possibly the GitHub one too — Eddie Jaoude, Feb 18 '16 at 16:01
@EddieJaoude yep, here's the [PR](https://github.com/womenhackfornonprofits/whfnp-content-api/pull/3) and [Travis project](https://travis-ci.org/womenhackfornonprofits/whfnp-content-api). NB: I understand why the test is failing (forgot to actually start the server before running test - d'oh!), but primary issue here is why CI is "greening" failed tests. Thanks! — krissy, Feb 18 '16 at 16:06
That is strange that the failing gulp test exited with exit code 0... "The command "gulp test" exited with 0." — Eddie Jaoude, Feb 18 '16 at 16:12

score 1 · Answer 1 · answered May 18 '16 at 13:50

This answer is not related to jasmin/frisby. But still might help other people looking on the gulp exiting with 0 on failing tests and having travis ci pass.

Here's my gulp test

gulp.task('test', function () {
    mochify('./src/js/__tests__/**/*.js', {
        extension: ['.jsx', '.js'],
        node: true,
        path: ['./src/js/'],
        watch: false,
    })
        .transform(babelify)
        .transform(envify({
            _: 'purge',
            NODE_ENV: 'production',
        }))
        .on('error', function(){
            // Exits gulp with error
            process.exit(1);
        }).bundle()
});

The part which I figured out was that you can manually exit gulp with an exit code non 0 via process.exit(1); and then travis will fail.

TravisCI passes despite tests failing, Gulp setup with tests on Jasmine + Frisby

1 Answers1