16

As much as I adore Cypress it's starting to turn out quite badly. I don't think I would be doing something fundamentally wrong. I've read best practices a couple of times and I don't see what I could improve really.

It's starting to be rather frustrating. Having tests working perfectly on the local machine (tried running several times in the row) and yet when the same code runs through the CI (currently Bitbucket Pipelines), several tests fail for weird reasons.

For example, a single click on an item in the list adds a single item in the cart. Works perfectly however I try to break it and yet the same test in CI makes that click happen twice there for some reason. And that's one issue I am at least able to describe. Others are out of whack and very often happening just randomly like that element is no visible and yet looking at the screenshot I can see just fine.

I tried to use cypress-failed-log plugin to see command log, but that does not really help because it's the same thing I see locally and yet it fails in CI. A video I can see in Cypress Dashboard is not that helpful either because it's usually way too fast and some things are not even seen there.

Can someone advise me some other options on how to approach the issue at hand more gracefully? I must admit I am on the verge of giving up on these tests because it just takes too much time to make them reliable.

Some details about my setup:

cypress-failed-log@2.5.0
cypress-testing-library@3.0.1
cypress@3.3.1

# job definition for running E2E tests in parallel
e2e: &e2e
  name: E2E tests
  image: cypress/browsers:chrome67-ff57
  caches:
    - yarn
    - home-cache
  script:
    - yarn -v
    - cd cypress
    - yarn install --frozen-lockfile
    - npx @bahmutov/print-env BITBUCKET
    - yarn ci --parallel --ci-build-id $BITBUCKET_BUILD_NUMBER
FredyC
  • 3,999
  • 4
  • 30
  • 38
  • We tried with dockers and it works smooths – N.. Jun 17 '19 at 07:44
  • I don't follow. You mean you have run tests in Docker? And how exactly did that help you to debug the tests flakiness? – FredyC Jun 17 '19 at 08:12
  • Yes we are running tests using dockers in build server. We configured for video in cypress.json that's allow use to see video on demand. – N.. Jun 17 '19 at 10:57
  • I've tried multiple versions of `Cypress`, in `headed`/`headless` mode and always the some case. Working completely fine locally, but breaks on `GitLab CI`... – Daniel Danielecki Mar 29 '21 at 11:23
  • @FredyC did you end up ditching cypress or what was your solution to the problem. I am facing similar issues. Locally it works fine but remotely inside gitlab ci it just breaks randomly. – Snake_py May 25 '22 at 08:34
  • @Snake_py Yea, as far I can remember, it was so frustrating that we decided to ignore C2E tests, it wasn't worth spending so much time on fixing these all the time. – FredyC Jun 09 '22 at 08:47

2 Answers2

1

Weird, do you use the same browser when you do your local test ? You use additional npm packages, maybe you could remove all the non essentials and retry ?

KMO
  • 57
  • 7
1

Cypress 3.3.0 claims to fix slow network requests. This could be relevant to CircleCI test failures.

In Cypress 3.1.4, @danielschwartz85 reported that they were seeing network slowness while making normal fetch requests. It was so bad that they reported load times of 5.5 seconds in Cypress, compared to 300ms in a regular browser - HTTP requests in Cypress were running up to 18 times slower than in Chrome for this test case. A large part of Cypress is that it should behave just like a normal web browser, so this was a major issue. Users expect Cypress to "behave like Chrome", not "behave like Chrome but 18x slower". So, we set out to investigate the source of this slowness.

https://docs.cypress.io/guides/references/changelog.html

GonzaSSH
  • 466
  • 3
  • 2
  • I suppose it could be one of the reasons, but definitely not all of them. If Cypress in CI clicks twice instead of once it can hardly be reasoned as network issues. – FredyC Jun 22 '19 at 07:26