Cypress: How to retry entire test suite?

Question

Goal Be able to perform multiple retries of an entire test suite until all tests succeed.

Instead of configuring retries allowed for every test within a test suite, as defined by Configure Test Retries: Global Configuration

How to provide test suite retries?

If any test fails within a test suite, then start over and retry the entire test suite again until all tests within a test suite succeed or stop when the entire test suite retry attempts exceed the requested retry attempts.

In the following code, configures retry attempts applicable for every test within a test suite.

The desired goal for this test suite, test 'TEST counter' is expected to fail until incrementing _counter equals _runModeRetries and only then is it expected to succeed. Repeat the entire test suite means re-run every test prior to test 'TEST counter' until it succeeds.

However, what happens is that only test 'TEST counter' is retried _runModeRetries times, and _counter is not incremented because test 'TEST increment' is called only once.

Why do I want this?

I have test suites that have a series of tests the required to run in sequence, and if a test fails, then retries require restarting the sequence. For example, _counter can only be incremented if test 'TEST increment' is called again with a full test suite retry.

How can I do test suite retries?

  let _counter = 0;
  const _runModeRetries = 3;

  context(
    'CONTEXT Cypress Retries',
    {
      retries: {
        runMode: _runModeRetries,
        openMode: 0
      }
    },
    () => {
      it('TEST increment', () => {
        _counter++;
        expect(_counter).to.be.a('number').gt(0);
      });

      it('TEST true', () => {
        expect(true).to.be.a('boolean').to.be.true;
      });

      it('TEST false', () => {
        expect(false).to.be.a('boolean').to.be.false;
      });

      it('TEST counter', () => {
        if (_counter < _runModeRetries) {
          assert.fail();
        } else {
          assert.isTrue(true);
        }
      });
    }
  );

Surely there is a better way to decouple your tests dependency. Things could get messy and hard to debug. — jjhelguero, May 11 '22 at 18:01

Fody · Answer 1 · 2022-05-10T23:55:03.530

This is really hacky, I'm but posting it in case you can improve on it

run the suite _runModeRetries times
add a skip variable to control if tests run
make all tests function() so that this.skip() can be called
add an after() within the suite to set skip true after first pass
add an onFail handler to reset skip when a fail occurs

let _counter = 0;
const _runModeRetries = 3;
let skip = false

Cypress.on('fail', (error, test) => {
  skip = false
  throw error  // behave as normal
})

Cypress._.times(_runModeRetries, () => {

  context('CONTEXT', {retries: {runMode: _runModeRetries}}, () => {
      it('TEST increment', function() {
        if (skip) this.skip()
        _counter++;
        expect(_counter).to.be.a('number').gt(0);
      });
      it('TEST true', function() {
        if (skip) this.skip()
        expect(true).to.be.a('boolean').to.be.true;
      });
      it('TEST false', function() {
        if (skip) this.skip()
        expect(false).to.be.a('boolean').to.be.false;
      });
      it('TEST counter', function() {
        if (skip) this.skip()
        assert.isTrue(_counter < _runModeRetries ? false :true);
      });
    }
  );
  after(() => skip = true)  // any fail, this does not run 
})

There may be some improvements available by adjusting the suite inside the onFail (suite = test.parent) and avoiding adding the changes to individual tests.

A cleaner way is to use the Module API which would allow you to run the test suite, examine the results and run again if there's any fail. Sort of manual-retry.

Thank you, I appreciate your reply! – Jeff May 11 '22 at 12:47 — Jeff, May 11 '22 at 12:47

Cypress: How to retry entire test suite?

1 Answers1