0

I have a section of code that runs if the user needs to re-auth after logging in. During UI tests, this popover is sometimes displayed, so I have a check for it existing

if (XCUIApplication().staticText["authLabel"].exists) {
    completeAuthDialog()
}

When this runs locally, it is fine, completes and the framework finds the element no problem. But when the nightly job on the CI is ran, it fails the first time, but once the same build is set to be rebuilt, the test passes. authLabel is the UILabel's accessibility identifier(btw), so I have been trying to figure out what is causing the flickering.

Yesterday I spent time on the issue, and it seems that the framework just doesn't find the elements sometimes? I have used the accessibility inspector in ensure I am query for the same time it sees.

I even expanded that if check with 4 or 5 additional || to check for any element inside of the popover. The Elements all have accessibility identifiers, I have also used the record feature to ensure that it passes back the same element "names" I am using.

I am kind of stuck, I don't know what else to try/could be causing this issue. The worst part is it ran fine for couple of months, but it seems to fail every night now, and as I said when the tests are ran locally inside xcode they pass fine. Could this be a issue with building from command line?

Jacob Boyd
  • 672
  • 3
  • 19
  • I also wanted to add, that I thought about/tried adding a observer for the notification that kicks off the popover, and using KVO, but the app and the testing framework run on separate processes from what I read from other posts here that are similar. Because the notification hook/observedValue function wasn't called. When the if check does fail locally, I can see the element on the screen (lol) – Jacob Boyd Sep 22 '17 at 15:05

1 Answers1

1

It is often slower when your tests execute on a different machine, this problem seems particularly prevelant with CI machines as they tend to be under-powered.

If you just do a single check for an element existing, the test only has one point in time to get it right and if the app was slow to present the element then the test will fail.

You can defend against having a flaky test by using a waiter to check a few times over a few seconds to ensure that you've given the app enough time to show the authentication dialog before continuing.

let authElement = XCUIApplication().staticText["authLabel"]
let existsPredicate = NSPredicate(format: "exists == true")
let expectation = XCTNSPredicateExpectation(predicate: existsPredicate, object: authElement)
let result = XCTWaiter().wait(for: [expectation], timeout: 5)
if (result == .completed) {
    completeAuthDialog()
}

You can adjust the timeout to suit your needs - a longer timeout will result in the test waiting a longer time to continue if the auth dialog doesn't appear, but will give the dialog more time to appear if the machine is slow. Try it out and see how flaky the tests are with different timeouts to optimise.

Oletha
  • 7,324
  • 1
  • 26
  • 46
  • I did not know about `XCTWaiter()` until after I posted this question. I will be updating my CI box to run xcode 8.3 next week. I think this will solve the issue, as using a `exists == true` predicate worked great when the dialog appears, but before `XCTWaiter()` if a expectation timed out the test would fail. Thank you for your answer, I should have updated my question :) – Jacob Boyd Sep 23 '17 at 23:00
  • Yeah, the test failing if a wait failed was so annoying! XCTWaiter is really helpful. :) – Oletha Sep 24 '17 at 07:51