How to automatically delete every failing Cypress test?

Question

I have inherited a large, out-of-date repo with a set of Cypress e2e tests that cumulatively take multiple days to run. I need to run the tests and then delete the ones that fail. Based on my experiences so far, around 80% of tests fail. So the scale of this task quickly becomes unwieldy manually.

Ideally the solution would be a single bash script. Another approach that comes to mind is to somehow export a list of failing tests to a CSV (which I haven't been able to figure out how to do either) and then delete every file in that list programmatically somehow. I'm using VSCode in case there are plugins that can help.

There is a secondary problem that simply running all tests quickly causes me to run out of memory. If there was some way to delete the tests as I go so that my entire task was accomplished by a single bash script, that would be amazing. But I can run the tests manually if this is not possible.

Right now I'm accessing the list of failed tests by just literally copying the terminal output to a text file. This would be easy enough to do programmatically, but the output doesn't even list the file names in an easy-to-extract manner. Example below (please overlook some weird formatting changes as I anonymized the file names for this post):

What is the best way to do this?

how do you know the test failed? is there a particular string in the output of the test that you can search for that would indicate failure (need to make sure said string does not show up in the output from a successfull test); does this 'fail' string come through stdout or stderr? is this 'fail' string written to an output/log file for you?; generally speaking ... capture all output (stdout/stderr) to a temp/scratch file, grep for 'fail' string and if found then delete (archive?) the test; be sure to save temp/scratch file to a permanent logfile (in case you need to review script's activity) — markp-fuso, Feb 04 '22 at 14:50
Right now I am just manually combing through what I literally copied and pasted from the terminal to a text file. — 223seneca, Feb 04 '22 at 14:55
I have updated the original question to indicate why just `grep`ing for a fail string isn't a full solution, given how the `stdout` content is formatted. — 223seneca, Feb 04 '22 at 15:04
Please update the question with an example of how you invoke a `cypress` test as well as what your command line/console is (eg, `bash`? windows cmd window? something else?) also update the question with sample `success` (aka non-fail) output; for the sample output you've currently provided ... how do you tell if it failed ... are you looking for the string 'failing' ... or do one of those characters on the first line (`1`, `-`) signify a fail? — markp-fuso, Feb 04 '22 at 15:04
The question as well as the tags already indicate that I am using `bash`. The `cypress` test is invoked with a `--testFile failingtest**` flag. A successful test is the same output except there is a checkmark where the `x` currently indicates a fail. — 223seneca, Feb 04 '22 at 15:11
thanks, wanted confirmation on the `bash` as some people apply the wrong tags; in the output you've shown the `X` looks like some sort of image and not necessarily an ascii `X` (or is that the font that makes it look like an image?); we need to see exactly what that output looks like; consider running `cypress --testFile failingtest** >> failingtest**.log 2>&1` (replace `**` as needed); update question with cut-n-paste (no images) of the entire `cypress` call plus the output from `cat failingtest**.log`, for both a 'fail' and a 'success' — markp-fuso, Feb 04 '22 at 15:26
what's in the image is not a capital X from the same font as the rest of the output; we need to see what the control codes are hence the need to capture the output to a file and then display the contents of the file; actually, probably need to see the output from `od -c failingtest**.log`; if the *only* way to distinguish between 'fail' and 'success' is the X/checkmark then we need to see the (control) codes for these characters so we know what to grep/search for — markp-fuso, Feb 04 '22 at 15:38
You said "around 80% of tests fail". Is it possible it will be less effort to just throw everything away and start from scratch? — SiKing, Feb 04 '22 at 18:15
I'd like to do that but it's been mandated otherwise. (This is a workplace task.) I'm continuing to push for throwing it all out, but honestly at this point I'm interested in this as an engineering problem on its own. It's not that weird of an edge case, but I can't find any existing solutions online. — 223seneca, Feb 04 '22 at 19:03

Fody · Accepted Answer · 2022-02-05T22:51:41.220

export a list of failing tests to a CSV - When you run the tests via Cypress module API, you are running them in a node script with access to each test result and fs to write the results out.

Here's the basic concept

// e2e-run-tests.js
const cypress = require('cypress')
const fs = require('fs')

cypress
  .run({
    // the path is relative to the current working directory
    spec: './cypress/integration/**/*.spec.js',
  })
  .then((results) => {
    console.log(results)
    const tests = results.runs[0].tests
    const fails = tests
      .filter(test => test.state === 'failed')
      .map(test => test.title.join(' - '))     // concat suite and test titles
    fs.writeFileSync('failed-tests.txt', fails)
  })
  .catch((err) => {
    console.error(err)
  })

Deleting tests automatically is like playing with a loaded gun.

Better, once you have the list you can prevent failures from running again with cypress-select-tests

// cypress/plugins/index.js

const selectTests = require('cypress-select-tests')

const failures = require('./failed-tests.txt')

// return test names you want to run
const pickTests = (filename, foundTests, cypressConfig) => {
  // found tests will be names of the tests found in "filename" spec
  // it is a list of names, each name an Array of strings
  // ['suite 1', 'suite 2', ..., 'test name']

  return foundTests.filter(fullTestName => {
    return !failures.includes(fullTestName)
  })
}

module.exports = (on, config) => {
  on('file:preprocessor', selectTests(config, pickTests))
}

This is a partial solution. I still need to search through a large repo with complicated file structure and delete the individual files programmatically. — 223seneca, Feb 05 '22 at 16:35

How to automatically delete every failing Cypress test?

1 Answers1