Keep getting an error when a test fails running tests on Android

Question

Description

DISCLAIMER: This was also posted on github here as an issue at the wix/Detox repo, so you can be better check the images there.

When running tests on an Android simulator, if one of them fails, at the end of the test report it is shown an error message like the following:

detox[8530] ERROR: [cli.js] Error: Command failed: node_modules/.bin/jest --config e2e/config.json '--testNamePattern=^((?!:ios:).)*$' --maxWorkers 1 "e2e"

If all tests pass, this message does not appears, as shown at the following link to the screenshot: detox-success

Below you can check the last part of what is shown in the terminal when a test fails:

Check device logs for full details!

      13 | 
      14 |   it("should have welcome screen", async () => {
    > 15 |     await expect(element(by.id("WelcomeScreen"))).toBeVisible()
         |                                                   ^
      16 |   })
      17 | 
      18 |   // it("should go to next screen after tap", async () => {

      at Client.execute (../node_modules/detox/src/client/Client.js:92:28)
      at InvocationManager.execute (../node_modules/detox/src/invoke.js:11:39)
      at MatcherAssertionInteraction.execute (../node_modules/detox/src/android/expect.js:128:35)
      at ExpectElement.toBeVisible (../node_modules/detox/src/android/expect.js:275:112)
      at _callee3$ (firstTest.spec.js:15:51)
      at tryCatch (../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.<computed> [as next] (../node_modules/regenerator-runtime/runtime.js:97:21)
      at tryCatch (../node_modules/regenerator-runtime/runtime.js:45:40)
      at invoke (../node_modules/regenerator-runtime/runtime.js:135:20)
      at ../node_modules/regenerator-runtime/runtime.js:170:11
      at callInvokeWithMethodAndArg (../node_modules/regenerator-runtime/runtime.js:169:16)
      at AsyncIterator.enqueue (../node_modules/regenerator-runtime/runtime.js:192:13)
      at AsyncIterator.prototype.<computed> [as next] (../node_modules/regenerator-runtime/runtime.js:97:21)
      at Object.exports.async (../node_modules/regenerator-runtime/runtime.js:216:14)

detox[9947] INFO:  at e2e/login-screen-tests/login-screen.spec.js:99:9 
detox[9947] INFO:  at e2e/login-screen-tests/login-screen.spec.js:100:9 
detox[9947] INFO:  [DetoxServer.js] server listening on localhost:36337...
 PASS  e2e/login-screen-tests/login-screen.spec.js (18.825s)

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 failed, 2 passed, 3 total
Snapshots:   0 total
Time:        36.553s
Ran all test suites matching /e2e/i with tests matching "^((?!:ios:).)*$".
detox[9939] ERROR: [cli.js] Error: Command failed: node_modules/.bin/jest --config e2e/config.json '--testNamePattern=^((?!:ios:).)*$' --maxWorkers 1 "e2e"

You can also see the full report at the screenshot in the link (I zoomed out a bit to fit on the screen): detox-error

And at the following link I'm showing the last pieces when running with detox test -c android.emu.debug --loglevel trace:

detox-error2

What could I be doing wrong? Can someone help me out figuring out what is the problem?

To Reproduce

• [X] I have tested this issue on the latest Detox release and it still reproduces

• package.json

{
  "name": "rmb-rn",
  "version": "0.5.0",
  "private": true,
  "scripts": {
    "android": "react-native run-android",
    "ios": "react-native run-ios",
    "start": "react-native start",
    "test": "jest",
    "lint": "eslint index.js app --fix --ext .js,.ts,.tsx",
    "test:e2e": "detox test -c ios.sim.debug",
    "build:e2e": "detox build -c ios.sim.debug",
    "ci:test:e2e": "detox test -c ios.sim.release -l verbose --cleanup",
    "ci:build:e2e": "detox build -c ios.sim.release",
    "compile": "tsc --noEmit -p . --pretty",
    "format": "npm-run-all format:*",
    "format:js": "prettier --write {.,**}/*.js",
    "format:json": "prettier --write {.,**}/*.json",
    "format:md": "prettier --write {.,**}/*.md",
    "format:ts": "prettier --write {.,**}/*.{ts,tsx}",
    "hack:types-react-navigation": "rimraf node_modules/@types/react-navigation/node_modules/@types",
    "hack:types-react-native": "rimraf node_modules/@types/react-native/node_modules/@types",
    "hack:types-react-test-renderer": "rimraf node_modules/@types/react-test-renderer/node_modules/@types",
    "patch": "patch-package",
    "postinstall": "node ./bin/postInstall",
    "prepare": "npm-run-all patch hack:*",
    "storybook": "(adb reverse tcp:7007 tcp:7007 || true) && start-storybook -p 7007",
    "build-storybook": "build-storybook -c .storybook -o .out",
    "adb": "adb reverse tcp:9090 tcp:9090 && adb reverse tcp:3000 tcp:3000 && adb reverse tcp:9001 tcp:9001 && adb reverse tcp:8081 tcp:8081"
  },
  "dependencies": {
    "@react-native-community/async-storage": "^1.5.1",
    "@react-native-community/datetimepicker": "^2.2.2",
    "@react-native-community/masked-view": "^0.1.7",
    "@storybook/addon-actions": "^6.0.0-alpha.2",
    "@storybook/addon-links": "^6.0.0-alpha.2",
    "@storybook/addons": "^6.0.0-alpha.2",
    "@storybook/react": "^6.0.0-alpha.2",
    "@types/node": "^13.5.3",
    "animated": "^0.2.2",
    "apisauce": "1.1.1",
    "babel-loader": "^8.0.6",
    "i18n-js": "^3.0.11",
    "lodash.throttle": "4.1.1",
    "mobx": "5.15.0",
    "mobx-react-lite": "^1.4.1",
    "mobx-state-tree": "^3.14.1",
    "moment": "^2.24.0",
    "native-base": "^2.13.8",
    "ramda": "0.26.1",
    "react": "16.12.0",
    "react-dom": "^16.12.0",
    "react-native": "0.61.5",
    "react-native-action-sheet": "^2.2.0",
    "react-native-camera": "^3.19.1",
    "react-native-contacts": "^5.0.7",
    "react-native-device-info": "^5.5.3",
    "react-native-gesture-handler": "^1.5.0",
    "react-native-keyboard-listener": "^1.1.0",
    "react-native-keychain": "4.0.1",
    "react-native-localize": "^1.0.0",
    "react-native-reanimated": "^1.4.0",
    "react-native-safe-area-context": "^0.7.3",
    "react-native-screens": "^2.0.0-alpha.6",
    "react-native-snap-carousel": "^3.8.4",
    "react-native-splash-screen": "3.2.0",
    "react-native-uuid": "^1.4.9",
    "react-native-vector-icons": "^6.6.0",
    "react-navigation": "4.2.2",
    "react-navigation-drawer": "^2.3.3",
    "react-navigation-stack": "2.2.3",
    "react-navigation-tabs": "^2.6.0",
    "reactotron-mst": "^3.1.1",
    "reactotron-react-native": "^4.0.0-beta.1",
    "validate.js": "0.13.1"
  },
  "devDependencies": {
    "@babel/core": "^7.7.4",
    "@babel/plugin-proposal-decorators": "^7.0.0",
    "@babel/plugin-proposal-optional-catch-binding": "^7.0.0",
    "@babel/runtime": "^7.7.4",
    "@emotion/core": "^10.0.15",
    "@storybook/addon-storyshots": "6.0.0-alpha.2",
    "@storybook/react-native": "^5.3.9",
    "@storybook/react-native-server": "^5.3.9",
    "@types/jest": "25.1.4",
    "@types/ramda": "0.26.36",
    "@types/react": "16.9.13",
    "@types/react-native": "0.60.23",
    "@types/react-navigation": "3.4.0",
    "@types/react-test-renderer": "16.9.1",
    "@typescript-eslint/eslint-plugin": "2.9.0",
    "@typescript-eslint/parser": "2.9.0",
    "babel-jest": "^25.2.3",
    "detox": "^16.0.1",
    "emotion-theming": "^10.0.14",
    "eslint": "^6.6.0",
    "eslint-config-prettier": "^6.0.0",
    "eslint-config-standard": "14.1.0",
    "eslint-plugin-import": "^2.15.0",
    "eslint-plugin-node": "10.0.0",
    "eslint-plugin-promise": "^4.0.1",
    "eslint-plugin-react": "^7.12.4",
    "eslint-plugin-react-native": "^3.6.0",
    "eslint-plugin-standard": "^4.0.0",
    "ignite-bowser": "^4.11.3",
    "jest": "^25.2.3",
    "jetifier": "^1.6.1",
    "metro-react-native-babel-preset": "^0.57.0",
    "npm-run-all": "4.1.5",
    "patch-package": "6.2.0",
    "postinstall-prepare": "1.0.1",
    "prettier": "1.19.1",
    "react-devtools-core": "4.2.1",
    "react-native-dotenv": "^0.2.0",
    "react-powerplug": "1.0.0",
    "react-test-renderer": "16.12.0",
    "rimraf": "3.0.0",
    "solidarity": "2.3.1",
    "typescript": "3.7.2"
  },
  "jest": {
    "preset": "react-native",
    "setupFiles": [
      "<rootDir>/node_modules/react-native/jest/setup.js",
      "<rootDir>/test/setup.ts"
    ],
    "testPathIgnorePatterns": [
      "/node_modules/",
      "/e2e"
    ],
    "transformIgnorePatterns": [
      "node_modules/(?!(jest-)?react-native|react-native|react-navigation|@react-navigation|@storybook|@react-native-community)"
    ]
  },
  "detox": {
    "test-runner": "jest",
    "configurations": {
      "ios.sim.debug": {
        "binaryPath": "ios/build/Build/Products/Debug-iphonesimulator/rmbRn.app",
        "build": "xcodebuild -workspace ios/rmbRn.xcworkspace -scheme rmbRn -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build -UseModernBuildSystem=NO",
        "type": "ios.simulator",
        "name": "iPhone 8"
      },
      "ios.sim.release": {
        "binaryPath": "ios/build/Build/Products/Release-iphonesimulator/rmbRn.app",
        "build": "xcodebuild -workspace ios/rmbRn.xcworkspace -scheme rmbRn -configuration Release -sdk iphonesimulator -derivedDataPath ios/build -UseModernBuildSystem=NO",
        "type": "ios.simulator",
        "name": "iPhone 8"
      },
      "android.emu.debug": {
        "binaryPath": "android/app/build/outputs/apk/debug/app-debug.apk",
        "build": "cd android && ./gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug && cd ..",
        "type": "android.emulator",
        "device": {
          "avdName": "Nexus_4_API_28"
        }
      },
      "android.emu.debug-real-device": {
        "binaryPath": "android/app/build/outputs/apk/debug/app-debug.apk",
        "build": "cd android && ./gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug && cd ..",
        "type": "android.attached",
        "name": "ce50806d386d"
      },
      "android.emu.release": {
        "binaryPath": "android/app/build/outputs/apk/release/app-release.apk",
        "build": "cd android && ./gradlew assembleRelease assembleAndroidTest -DtestBuildType=release && cd ..",
        "type": "android.emulator",
        "device": {
          "avdName": "Nexus_4_API_28"
        }
      }
    }
  }
}

• e2e folder below:

• 1) init.js

const detox = require("detox")
const config = require("../package.json").detox
const adapter = require("detox/runners/jest/adapter")

jest.setTimeout(120000)
jasmine.getEnv().addReporter(adapter)

beforeAll(async () => {
  await detox.init(config)
})

beforeEach(async () => {
  await adapter.beforeEach()
})

afterAll(async () => {
  await adapter.afterAll()
  await detox.cleanup()
})

• 2) config.json

{
  "setupFilesAfterEnv": ["./init.js"],
  "testEnvironment": "node"
}

• 3) firstTest.spec.js The test suite with a failing test

{
  describe("Example", () => {
  beforeEach(async () => {
    await device.reloadReactNative()
  })

  it('should be visible the account number', async () => {
    const usernameInput = await element(by.id("loginScreen-accountNumber"))
    await expect(usernameInput).toBeVisible()
  })

 // the failing test is below
  it("should have welcome screen", async () => {
    await expect(element(by.id("WelcomeScreen"))).toBeVisible()
  })
})
}

Expected behavior

The expected behavior, I believe, is the test report to end without displaying an error.

Device Logs

adb_logs.log

Environment:

• Detox: 16.0.1

• React Native: 0.61.5

• Node: 12.16.1

• Device: AVD Android Studio Nexus 4 API 28

• OS: Ubuntu 18.04.4 LTS

• Test-runner (select one): Jest (as shown in the package.json above)

Answer 1

The error is perfectly legitimate, pointing out that Detox works as expected.

When it comes to test orchestration and execution, Detox is in fact a mere wrapper around an actual test runner -- Jest in your case (the better choice IMO :). That fact is mentioned in the setup guide and explained more elaborately in the more specific guide for setting up Jest:

As already mentioned in the Getting Started guide, Detox itself does not effectively run tests logic, but rather delegates that responsibility onto a test runner. 
Jest is the recommended runner for projects with test suites that have become large enough so as to require parallel execution.

In turn, that also affects what the Detox CLI does - which is to simply execute the test runner with proper arguments. So, to conclude, when all tests pass, Jest ends successfully, and silently. When something fails, Jest logs an error, gives clues as to what caused it, tells everyone it has failed, and returns a non-0 value. This is all coherent with what you've described.