Is there a way to continue test scenario execution after step failure in a previous scenario?

Question

Whenever there is a step failure while running on a remote server, I would like to capture the failed step and then continue running the remaining scenarios. The captured step would then be included in a file for reporting purposes. Is this a possibility? All replies I've seen elsewhere just say you should fix the test before moving on. I agree, but I only want the tests to stop when running locally, not remotely.

➜ customer git:(pat104) ✗ cucumber.js -f progress (pat104⚡) ...F-----Failed scenario: View and select first contact from contact history ...F-Failed scenario: View and select a contact from multiple contacts in history ..................................................F---Failed scenario: Navigating to profile with url and enrollmentId ...................................................F-Failed scenario: Successful MDN Search with 1 result returned. Tech Selects and continues .............FFailed scenario: Successful MDN with multiple results

Can you add the command you are using to run the test && the output you are getting? What are you using for reports? — John Dhom, Jul 30 '15 at 20:09

score 3 · Accepted Answer · answered Aug 03 '15 at 20:22

Turns out, one of the step-definitions was using .waitForExist incorrectly. The test was written:

this.browser
        .waitForExist('#someElement', 1000, callback)

Callback isn't a parameter for .waitForExist, rewrote to:

.waitForExist('#someElement',1000).then(function (exists) {
            assert.equal(exists, true, callback);
        })

score 0 · Answer 2 · answered Jul 30 '15 at 20:09

0

This is the default behavior, isn't it? Example command

cucumber.js -f json > cucumberOutput.json

answered Jul 30 '15 at 20:09

John Dhom

1,022
9
21

No, if I run the tests with `cucumber-js -f progress` the step still breaks the test and doesn't continue. – Jessica Wood Jul 30 '15 at 20:17
This is key as test should always continue to run. – John Dhom Aug 03 '15 at 19:40
I now have a test that fails, which stops cucumber execution, and no json is produced at all :( – John Dhom Aug 03 '15 at 20:15

nilesh · Answer 3 · 2015-08-01T23:11:59.327

Well, that you need to manage in your test itself using callback.fail(e) like below. You can use library like grunt-cucumberjs to add these errors to nice HTML reports.

this.Then(/^the save to wallet button reflects the offer is saved$/, function (callback) {
        merchantPage(this.nemo).doStuff().then(function () {
            callback();
        }, function (e) {
            callback.fail(e); //add this to report
        });
});

Or you could use Hooks and check whether a scenario is failed and report (take screenshot or add logging etc.)

 this.After(function (scenario, callback) {
        var driver = this.nemo.driver;
        if(scenario.isFailed()){
            driver.takeScreenshot().then(function (buffer) {
                scenario.attach(new Buffer(buffer, 'base64').toString('binary'), 'image/png');
            });
        }
        driver.quit().then(function () {
                callback();
       });
 });

After adding this to the test, it still skips the remaining scenarios in the feature after a step failure. Also, having to add this to every test seems pretty messy. — Jessica Wood, Jul 31 '15 at 18:36
[one of] the issues is that her tests are stopping even though there is no .quit() in the .After() hook. Have you seen this when there are exceptions? I think I have. — John Dhom, Aug 03 '15 at 19:41

Is there a way to continue test scenario execution after step failure in a previous scenario?

3 Answers3