2

In order to make future changes easier, we have put the login script in a page object.

//Login.js

export class Login {

username_input=         () => cy.get('#LoginForm_username');
password_input=         () => cy.get('#LoginForm_password');
login_button=           () => cy.contains('Login');
profile=                () => cy.get('.profile-content');

login(email, password){
    this.username_input().type(email);
    this.password_input().type(password);
    this.login_button().click();
    this.profile().should('exist')
        return this;
    }
}

So later we can just reuse it in any spec file

//actual.spec.js

import {Login} from "../../pages/login/Login";

it('logs in', () => {
login.login(Cypress.env('userEmail'), Cypress.env('userPass'))
}

Now our login page has the strange behavior of sometimes not responding. Therefore we added retries.

We have found two ways that work, but both are not ideal:

Solution 1:

Put retries param in the config file

"retries": 2

Why this is not ideal?

This enables retries for every single test, which we don't want. We only want it for the login script.

Solution 2:

Put the retry param in the 'it'

import {Login} from "../../pages/login/Login";

it('logs in', {retries:2} () => {
login.login(Cypress.env('userEmail'), Cypress.env('userPass'))
}

Why this is not ideal?

We have to put the param in every spec file and if we want to change the number of retries or get rid of retries entirely, we need to change it in every single spec file.

Solution 3???

What I am looking for now is a way to put the retry param somewhere in the login functionality in the login.js but I could not find a way to do that.

cypher_null
  • 632
  • 8
  • 22

3 Answers3

2

Some more ideas:

On-the-fly retry changes

TLDR
Upside: simple
Downside: uses internal commands

There's an internal command that will allow you to change the retry count mid-test.

I stress the word internal as a warning that it may stop working at some new release.

With your code

export class Login {
  ...

  login(email, password) {

    const originalRetries = cy.state('runnable')._retries

    cy.state('runnable')._retries = 5    // WARNING - internal command
    
    this.username_input().type(email);
    ...

    cy.state('runnable')._retries = originalRetries  // restore configured retries value
  }
}

Retry by recursion

TLDR
Upside: good for flaky situations
Downside: explicit wait required

If you want the longer but more mainstream solution, use recursion.

You will need a non-failing check to handle the retry-or-finish logic, which will essentially mean changing cy.get('.profile-content').should('exist') to a jQuery equivalent and using an explicit wait (a downside for this pattern).

export class Login {
  ...

  loginAttempt(attempt = 0) {       
    
    if (attempt === 3) throw 'Unable to login after 3 attempts'

    this.login_button().click();
    
    // wait for login actions to change the page
    cy.wait(200).then(() => {  
      
      // test for success
      if (Cypress.$('.profile-content').length > 0) {
        return // found it, so finish
      }

      // retry
      loginAttempt(email, password, ++attempt)  
    })
  }

  login(email, password) {
    this.username_input().type(email);  
    this.password_input().type(password);
    loginAttempt()
    return this
  }
}

You can cut the cy.wait(200) down a bit and raise the maximum attempt, say cy.wait(20) and if (attempt === 300)

The downside is each retry uses up a bit more heap memory, so taking that too far will be detrimental - you'll need to experiment.


Set logged-in state directly

TLDR
Upside: By-passes flaky parts of test
Downside: Requires knowledge of login state

The other aspect you may consider is to cut out the flaky login altogether. To do that, you need to find out what the app considers to be logged-in. Is it a cookie, localstorage value, etc.

For each test that needs to be in logged-in state but is not actually testing the login process, set that state directly.

An example is given here recipes - logging-in__using-app-code

describe('logs in', () => {
  it('by using application service', () => {
    cy.log('user service login')

    // see https://on.cypress.io/wrap
    // cy.wrap(user promise) forces the test commands to wait until
    // the user promise resolves. We also don't want to log empty "wrap {}"
    // to the command log, since we already logged a good message right above
    cy.wrap(userService.login(Cypress.env('username'), Cypress.env('password')), {
      log: false,
    }).then((user) => {
    // the userService.login resolves with "user" object
    // and we can assert its values inside .then()

      // confirm general shape of the object
      expect(user).to.be.an('object')
      expect(user).to.have.keys([
        'firstName',
        'lastName',
        'username',
        'id',
        'token',
      ])

      // we don't know the token or id, but we know the expected names
      expect(user).to.contain({
        username: 'test',
        firstName: 'Test',
        lastName: 'User',
      })
    })
function login (username, password) {
  const requestOptions = {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ username, password }),
  }

  return fetch(`${config.apiUrl}/users/authenticate`, requestOptions)
  .then(handleResponse)
  .then((user) => {
    // login successful if there's a jwt token in the response
    if (user.token) {
      // store user details and jwt token in local storage to keep user logged in between page refreshes
      localStorage.setItem('user', JSON.stringify(user))
    }

    return user
  })
}
Fody
  • 23,754
  • 3
  • 20
  • 37
  • This is simply amazing! Thanks for the extremely detailed breakdown of the different options. I will immediately start experimenting with this 3 solutions and see if I can get one or more of them to work. Ideally I can get it to work to set the log-in state directly. I will keep you updated :) – cypher_null Oct 14 '21 at 06:18
  • On-the-fly retries works perfectly! I am just a bit worried by your remark about it maybe stopping to work a new release. Retry by recursion gives me a hard time. It seems to do nothing but click. It bypasses the cy.wait() and the email and PW input and just returns 3 failed attempts. Increasing the cy.wait() did nothing. Unfortunately it outputs almost no report except that attempt 1,2,3, failed. Need to look more into this. Still working on the last solution, but will need a dev to help me out there – cypher_null Oct 14 '21 at 07:27
  • I made a mistake in the recursion - have adjusted the code after seeing another answer. – Fody Oct 14 '21 at 23:57
  • Personally I'd use `cy.state('runnable')._retries`, but I have to attach the warning for anybody reading this code. Hopefully Cypress will add it as an official recipe, as you have identified a gap in the existing retry patterns. – Fody Oct 15 '21 at 00:01
  • Yes, this is the solution I currently use. It works just as I need it to. I will make a comment with a warning in the code. – cypher_null Oct 15 '21 at 05:57
0

(Editing the post to include the 2nd workaround...)

The entire idea of cypress is avoiding flaky test. If you have a login that sometimes fails and sometimes doesn't, then you have an unstable environment even before you start.

  1. If all your tests depend on the login then it might not be a bad idea to include the retry at suite level (in the describe) but if only some of your tests rely on the login then you could group them all in only one suite and then add the retry at suite level.

Something like..


describe('Your suite name', {
  retries: {
    runMode: 2,
    openMode: 2,
  }
}, () => {
 
//number of retries will be applied per test
  it('logs in', () => {
    login.login(Cypress.env('userEmail'), Cypress.env('userPass'))
  })
  
  it('other test', () => {
})

})
  1. Do the login through API and that one make it part of the tests that rely on the login (so it doesn't have to go through the UI and you won't get the issue). When testing the login itself then only there, do the retry for that test.
Vicko
  • 234
  • 4
  • 17
  • Moving the retry from the it('logs in') to the describe('') part is basically a mix of solution #1 and solution #2 with the drawbacks of both. This would mean that we still need to put the retries in every spec file (and change it in every single one if needed) and it now retries all tests in the suite which is also undesireable. – cypher_null Oct 13 '21 at 11:49
  • 1
    There's one more solution for it (but you will not be testing the login UI every time) which is logging through API and then continue with your tests and only have the UI for login tested in one spec only. That will mean you add the number of retries only for that spec and the rest should work. Unless your problem is API related ant that's bigger – Vicko Oct 13 '21 at 12:03
  • That might actually be the best idea! Need to look into logging in via POST again. We kinda gave it up as we couldn't get around CSRF login yet. Tried all the scripts from https://github.com/cypress-io/cypress-example-recipes/tree/master/examples/logging-in__csrf-tokens but none worked. But that is for another topic. Guess this solution is the best one – cypher_null Oct 13 '21 at 12:13
  • Glad I could help :) – Vicko Oct 13 '21 at 12:42
0

When you write:

Now our login page has the strange behavior of sometimes not responding.

You can rest assured, that your tests are not the problem.

Martin Zeitler
  • 1
  • 19
  • 155
  • 216