Running multiple cypress instances locally results in unpredictable errors

Question

We have a beefy server in our CI and we want to take advantage of it and parallelize our cypress test suite on the same machine. We know that cypress doesn't encourage it but it should be possible!

We have a bash script that splits all of the test files in n groups and runs cypress on each group on a new port in the background with:

npx cypress run --spec $specFiles --port $port --headless &

In principle, this should work as each process will run its files on a separate headless browser. However, with >4 workers, we get all sort of errors:

Custom commands not found
Tests that pass on a single worker now fail non-deterministically:
Syntax errors?!? (code runs correctly on a single worker)
Timeouts - these shouldn't happen since the webserver is not under any significant load!

We are trying our best to avoid running each cypress instance as a new docker container to avoid the additional CI complexity but will dive into that if necessary. Are we missing something obvious here?

Here is the complete script for reference:

#!/bin/bash

nThreads=3

print_usage() {
  printf "Usage:
    ./run_tests_parallel.sh -n <number of threads to use>
    Defaults to $nThreads threads\n"
  exit 0;
}

while true; do
  case "$1" in
    -n | --threads ) nThreads=$2; shift 2 ;;
    -h | --help ) print_usage ; shift ;;
    -- ) shift; break ;;
    * ) break ;;
  esac
done

echo Using $nThreads threads

# Return non-zero if any of the subprocesses
# returns non-zero
set -eu

testFiles=`find . -name "*.test.ts" -not -path "./node_modules/*"`

# init testF
testFilesPerThread=()
for (( n=0; n<$nThreads; n++ )); do
    testFilesPerThread+=("")
done

i=0
for testFile in $testFiles; do
    testFilesPerThread[$i]="${testFilesPerThread[$i]} $testFile"
    i=$((($i + 1)%$nThreads))
done

pids=()
for (( i=0; i<${#testFilesPerThread[@]}; i++ )); do
    echo Thread $i has files: ${testFilesPerThread[$i]}
    # strip string and join files with ","
    specFiles=`echo ${testFilesPerThread[$i]} | xargs | tr -s "\ " ","`
    port=$((30001+$i))

    # run tests in background
    npx cypress run --spec $specFiles --port $port --headless &
    pids+=($!)
    echo "Spawned PID ${pids[${#pids[@]}-1]} for thread $i on port $port"
done


for pid in ${pids[@]} ; do
  echo "Waiting for PID $pid."
  wait $pid
done

echo DONE.

Fody · Answer 1 · 2021-10-13T10:13:42.590

I'm not an expert, but "Unexpected end of input" sounds like a file access clash has happened. Perhaps two processes have attempted to write to the same test artefact.

I heard that generally the number of threads should not exceed the number of cores - 1. On my 4 core machine, specifying 3 threads gets me about 15% increase in throughput over 20 specs.

I've used a NodeJS script to call the Cypress Module API, which allows adjustment of config on a per-thread basis to avoid file write clashes (see reporterOptions)

const path = require('path')
const fs = require('fs-extra')
const cypress = require('cypress')

const walkSync = function(dir, filelist = []) {
  const files = fs.readdirSync(dir);
  files.forEach(function(item) {
    const itemPath = path.join(dir, item)
    if (fs.statSync(itemPath).isDirectory()) {
      filelist = walkSync(itemPath, filelist);
    }
    else {
      filelist.push(itemPath);
    }
  });
  return filelist;
};
const files = walkSync(path.join(__dirname, '../cypress/integration'))

const groups= 3
const groupSize = Math.ceil(files.length / groups)
const groups = files.reduce((acc, file) => {
  if (!acc.length || acc[acc.length-1].length === groupSize) acc.push([]);
  acc[acc.length-1].push(file)
  return acc
},[])

console.time('process-time')
const promises = groups.map((group, i) => {
  return cypress.run({
    config: {
      video: false,
      screenshotsFolder: `cypress/screenshots/group${i}`,
      reporterOptions: {
        mochaFile: `results/group${i}/my-test-output.xml`,
      }
    },
    spec: group.join(','),
  })
})

Promise.all(promises).then(results => {
  console.timeEnd('process-time')
});

Please also see Is there any way to run parallel cypress tests on local machine which uses cypress-parallel.

score 0 · Accepted Answer · answered Oct 13 '21 at 20:41

As mentioned by Fody in this answer:

sounds like a file access clash has happened

All those errors were from a race condition caused by each individual background process transpiling our typescript code. Sometimes a cypress instance would try to run the tests while another process was transpiling the code and that would cause all sorts of errors including the weird Syntax error.

We've decided to run each group of tests in its own container for now as that's the only way to isolate each test suite in its own environment

Running multiple cypress instances locally results in unpredictable errors

2 Answers2