iOS tests fails in Release but works in Debug

Question

I just did a small set of tests for my react-native app and they work well when testing in iOS - Debug mode. But when I try to run it in iOS - Release it just fails, saying can't find the components.

I know more information is needed but I have no idea how to actually move forward. Notes: - All testID are in <View /> - I tried to run with simulator open and the app actually opens - Did trace and it has no much useful data (but the hierarchy shows no accessibilityID or testID)

Follows the hierarchy:

        Cannot find UI Element.
        Exception with Assertion: {
          "Assertion Criteria":  "assertWithMatcher:matcherForSufficientlyVisible(>=0.750000)",
          "Element Matcher":  "((!(kindOfClass('RCTScrollView')) && (respondsToSelector(accessibilityIdentifier) && accessibilityID('WelcomeScreen_Contents'))) || (((kindOfClass('UIView') || respondsToSelector(accessibilityContainer)) && parentThatMatches(kindOfClass('RCTScrollView'))) && ((kindOfClass('UIView') || respondsToSelector(accessibilityContainer)) && parentThatMatches((respondsToSelector(accessibilityIdentifier) && accessibilityID('WelcomeScreen_Contents'))))))",
          "Recovery Suggestion":  "Check if the element exists in the UI hierarchy printed below. If it exists, adjust the matcher so that it accurately matches element."
        }


        Error Trace: [
          {
            "Description":  "Interaction cannot continue because the desired element was not found.",
            "Error Domain":  "com.google.earlgrey.ElementInteractionErrorDomain",
            "Error Code":  "0",
            "File Name":  "GREYElementInteraction.m",
            "Function Name":  "-[GREYElementInteraction matchedElementsWithTimeout:error:]",
            "Line":  "124"
          }
        ]

        Hierarchy: <UIWindow:0x7fb2a5622f30; AX=N; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |--<RCTRootView:0x7fb2a5620e70; AX=N; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |--<RCTRootContentView:0x7fb2a57c5200; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |--<RCTView:0x7fb2a558c7a0; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |--<RCTView:0x7fb2a55baaf0; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |--<RCTView:0x7fb2a55a6f50; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |--<RCTView:0x7fb2a5517b40; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |--<RCTView:0x7fb2a5791380; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |--<RCTView:0x7fb2a5791070; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a5790d60; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a5790a50; AX=N; AX.frame={{0, 0}, {414, 88}}; AX.activationPoint={207, 44}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 88}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a5790740; AX=N; AX.frame={{-414, 0}, {414, 88}}; AX.activationPoint={-207, 44}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{-414, 0}, {414, 88}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a579d570; AX=N; AX.frame={{-414, 0}, {414, 88}}; AX.activationPoint={-207, 44}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 88}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a579d260; AX=N; AX.frame={{-414, 44}, {414, 43.5}}; AX.activationPoint={-207, 65.75}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 44}, {414, 43.5}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a578a1b0; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a5593d10; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55a5c90; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55d0c60; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 0}, {414, 896}}; AX.activationPoint={207, 448}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 896}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55d92f0; AX=N; AX.label='Welcome! Let's go! Sign in'; AX.frame={{0, 44}, {414, 818}}; AX.activationPoint={207, 453}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 44}, {414, 818}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55d0950; AX=N; AX.label='Sign in'; AX.frame={{0, 816}, {414, 36}}; AX.activationPoint={207, 834}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 772}, {414, 36}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cd670; AX=Y; AX.id='Touchable'; AX.label='Sign in'; AX.frame={{0, 816}, {414, 36}}; AX.activationPoint={207, 834}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 36}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTTextView:0x7fb2a55b3040; AX=Y; AX.label='Sign in'; AX.frame={{0, 816}, {414, 36.5}}; AX.activationPoint={207, 834.25}; AX.traits='UIAccessibilityTraitStaticText'; AX.focused='N'; frame={{0, 0}, {414, 36.5}}; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cd360; AX=N; AX.label='Welcome! Let's go!'; AX.frame={{0, 44}, {414, 772}}; AX.activationPoint={207, 430}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {414, 772}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cd050; AX=N; AX.frame={{207, 500}, {0, 316}}; AX.activationPoint={207, 658}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{207, 456}, {0, 316}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55ccd40; AX=N; AX.label='Let's go!'; AX.frame={{147, 460}, {120, 40}}; AX.activationPoint={207, 480}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{147, 416}, {120, 40}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cca30; AX=Y; AX.id='Touchable'; AX.label='Let's go!'; AX.frame={{147, 460}, {120, 40}}; AX.activationPoint={207, 480}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 0}, {120, 40}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTTextView:0x7fb2a55cc800; AX=Y; AX.label='Let's go!'; AX.frame={{164, 471}, {86, 18.5}}; AX.activationPoint={207, 480.25}; AX.traits='UIAccessibilityTraitStaticText'; AX.focused='N'; frame={{17, 11}, {86, 18.5}}; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTTextView:0x7fb2a55cc210; AX=Y; AX.label='Answer two quick questions to get started.'; AX.frame={{68, 379.5}, {278, 50.5}}; AX.activationPoint={207, 404.75}; AX.traits='UIAccessibilityTraitStaticText'; AX.focused='N'; frame={{68, 335.5}, {278, 50.5}}; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cbd20; AX=N; AX.label='Welcome to '; AX.frame={{97.5, 44}, {219, 336}}; AX.activationPoint={207, 212}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{97.5, 0}, {219, 336}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTView:0x7fb2a55cba10; AX=N; AX.label='Welcome to '; AX.frame={{97.5, 327}, {219, 33}}; AX.activationPoint={207, 343.5}; AX.traits='UIAccessibilityTraitNone'; AX.focused='N'; frame={{0, 283}, {219, 33}}; opaque; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTImageView:0x7fb2a55cb730; AX=N; AX.id='/Users/.../Library/Developer/CoreSimulator/Devices/4FF22CAE-8C08-4FA8-8CB7-DEC31EB77737/data/Containers/Data/Application/608DC539-4BB6-4C2D-934E-98C4863ED4D1/Library/Application Support/CodePush/76158f1b9b351e2881c905cc31e6f30ea33a2e52829f92e0b092dcc45be9cfde/CodePush/assets/app/assets/images/logo@2x.png'; AX.frame={{258.5, 333}, {58, 21}}; AX.activationPoint={287.5, 343.5}; AX.traits='UIAccessibilityTraitImage'; AX.focused='N'; frame={{161, 6}, {58, 21}}; alpha=1>
          |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |--<RCTTextView:0x7fb2a55cb500; AX=Y; AX.label='Welcome to '; AX.frame={{97.5, 327}, {162, 33.5}}; AX.activationPoint={178.5, 343.75}; AX.traits='UIAccessibilityTraitStaticText'; AX.focused='N'; frame={{0, 0}, {162, 33.5}}; alpha=1>

           9 |
          10 |   it('WelcomeScreen: Loaded', async () => {
        > 11 |     await expect(element(by.id('WelcomeScreen_Contents'))).toBeVisible();
             |                                                            ^
          12 |   });

I also tried some other values:

        <View
          style={styles.contents}
          testID="WelcomeScreen_Contents1"
          accessibilityHint="WelcomeScreen_Contents2"
          accessibilityLabel="WelcomeScreen_Contents3"
        >

And the following tests:

  it('WelcomeScreen: Loaded 1', async () => {
    await expect(element(by.id('WelcomeScreen_Contents1'))).toBeVisible();
  });

  it('WelcomeScreen: Loaded 2', async () => {
    await expect(element(by.id('WelcomeScreen_Contents2'))).toBeVisible();
  });

  it('WelcomeScreen: Loaded 3', async () => {
    await expect(element(by.id('WelcomeScreen_Contents3'))).toBeVisible();
  });

Detox config at package.json

  "detox": {
    "configurations": {
      "iphonexr": {
        "binaryPath": "ios/build/xxx/Build/Products/Release-iphonesimulator/xxx.app",
        "build": "RCT_NO_LAUNCH_PACKAGER=1 xcodebuild -quiet -workspace ios/xxx.xcworkspace -UseNewBuildSystem=NO -configuration Release -scheme xxx -derivedDataPath ios/build/xxx -sdk iphonesimulator",
        "type": "ios.simulator",
        "name": "iPhone Xʀ"
      },
      "iphonese": {
        "binaryPath": "ios/build/xxx/Build/Products/Release-iphonesimulator/xxx.app",
        "build": "RCT_NO_LAUNCH_PACKAGER=1 xcodebuild -quiet -workspace ios/xxx.xcworkspace -UseNewBuildSystem=NO -configuration Release -scheme xxx -derivedDataPath ios/build/xxx -sdk iphonesimulator",
        "type": "ios.simulator",
        "name": "iPhone SE"
      }
    },
    "test-runner": "jest"
  }
  }

@vk.4884 sorry the delay, just updated the information. The problem seems to be a little intermittent... — SidFerreira, Aug 05 '19 at 19:44
It's weird because I tried "by.label" too and it also fails! — SidFerreira, Aug 05 '19 at 19:53
Take a look at your hierarchy. None of the accessibility values exist there. That means that either they are being optimized out by RN or somehow you have a bug in the app. — Léo Natan, Aug 05 '19 at 21:09
@LeoNatan I wonder if I'm using some sort of flag that actually clean them up — SidFerreira, Aug 05 '19 at 21:54
Could be also that RN is doing some view flattening: https://twitter.com/shergin/status/1058393187079704576 A "view" is not something you would normally test for. Perhaps try testing the content, rather than the wrapper view. — Léo Natan, Aug 05 '19 at 22:00
The views I'm testing actually do some layout... still trying to find when exactly it stops working — SidFerreira, Aug 05 '19 at 22:05
The problem seems to be related to View Flattening indeed... But even adding a lot of extra A11Y labels and IDs, in many components, RN seems to actually ignore them! One example is that ignores my testID and picks the image name instead... — SidFerreira, Aug 06 '19 at 09:59
@LeoNatan, maybe you want to add this one to the docs as a Troubleshooting: It was CodePush!!! Please, feel free to answer my question so I can properly reward you. — SidFerreira, Aug 06 '19 at 10:22
Detox docs are not meant to be app developer support, especially for non-standard features. You can answer the question and I’ll upvote. — Léo Natan, Aug 06 '19 at 10:24

score 3 · Answer 1 · answered Aug 06 '19 at 10:39

3

After hours of debugging I found out the problems were: - CodePush overriding my changes - RN's View Flattening ( https://twitter.com/shergin/status/1058393187079704576 )

I added a small check so I can disable CodePush for Test builds and it works properly now.

answered Aug 06 '19 at 10:39

SidFerreira

551
3
20

1

Be aware that Apple has been cracking down from time to time on apps using CodePush and similar technologies. I'd recommend transitioning out of using it. – Léo Natan Aug 06 '19 at 19:53
@LeoNatan I'm not sure where you got that information but it is incorrect. CodePush and other interpreted code deployment technologies are explicitly allowed by section 3.3.2 of Apple's developer agreement. – imagio Feb 15 '20 at 01:40
1

Could you post your small check and how you disabled codepush for test builds? – BoKKeR Mar 12 '20 at 12:42
I've created a `CodePushify` component. And on E2E it doesn't add the CodePush. @BoKKeR – SidFerreira Mar 13 '20 at 20:03

iOS tests fails in Release but works in Debug

1 Answers1