Cryptic NSInternalInconsistencyException when running unit tests in Xcode 9 GM

Question

I'm running my iOS app's unit tests on Xcode 9 GM, and a couple of them are failing with a weird NSInternalInconsistencyException, complaining that some test assertions cannot be reported, because the implicated tests have no associated XCTestRun object. I'm using OCMockito + OCHamcrest for mocking and call verification.

For demonstration purposes, let's say my app is called MyTestApp, and I have a test class FooTest (which inherits from XCTestCase). In -setUp, I create and wire together the various mock objects for the tests, and in -tearDown I set them to nil just in case.

Here's an example of the exception I'm getting:

2017-09-19 13:23:01.852729-0700 xctest[17006:5392130] *** Assertion failure in -[FooTest recordFailureWithDescription:inFile:atLine:expected:], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-13201/Sources/XCTestFramework/Core/XCTestCase.m:308
/Users/fooUser/Workspaces/MyTestApp/tests/FooTest.mm:46: error: -[FooTest testSomething] : failed: caught "NSInternalInconsistencyException", "Unable to report test assertion failure 'Expected 1 matching invocation, but received 0' from /Users/fooUser/Workspaces/MyTestApp/tests/FooTest.mm:135 because it was raised inside test case -[FooTest testSomethingElse] which has no associated XCTestRun object. This may happen when test cases are constructed and invoked independently of standard XCTest infrastructure."
(
    0   CoreFoundation                      0x000000010255d1cb __exceptionPreprocess + 171
    1   libobjc.A.dylib                     0x0000000101ebff41 objc_exception_throw + 48
    2   CoreFoundation                      0x0000000102562362 +[NSException raise:format:arguments:] + 98
    3   Foundation                          0x0000000101827089 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 193
    4   XCTest                              0x0000000101d96875 -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:] + 518
    5   MyAppUnitTests                     0x000000011d7f2f5a -[HCXCTestFailureReporter executeHandlingOfFailure:] + 154
    6   MyAppUnitTests                     0x000000011d7f2b73 -[HCTestFailureReporter handleFailure:] + 67
    7   MyAppUnitTests                     0x000000011d64ec9a MKTFailTest + 217
    8   MyAppUnitTests                     0x000000011d64c3f4 -[MKTExactTimes verifyData:] + 254
    9   MyAppUnitTests                     0x000000011d64f2ba -[MKTBaseMockObject verifyInvocation:usingVerificationMode:] + 137
    10  MyAppUnitTests                     0x000000011d64f20b -[MKTBaseMockObject handlingVerifyOfInvocation:] + 115
    11  MyAppUnitTests                     0x000000011d64f15a -[MKTBaseMockObject forwardInvocation:] + 64
    12  CoreFoundation                      0x00000001024dfed8 ___forwarding___ + 760
    13  CoreFoundation                      0x00000001024dfb58 _CF_forwarding_prep_0 + 120
    14  MyAppUnitTests                     0x000000011c40b486 -[FooTest setUp] + 294
    15  XCTest                              0x0000000101d97b39 __24-[XCTestCase invokeTest]_block_invoke_3 + 31
    16  XCTest                              0x0000000101d97809 __24-[XCTestCase invokeTest]_block_invoke + 271
    17  XCTest                              0x0000000101ddff45 -[XCUITestContext performInScope:] + 183
    18  XCTest                              0x0000000101d976ef -[XCTestCase invokeTest] + 141

Several things of note:

The unit tests are not initializing XCTestCase or any XCTest "infrastructure" classes outside of the XCTest framework.
The remaining statements of the affected unit tests actually gets executed, up to the point of the exception.
The implicated line of code looks like: [verify(_mockTestObject) description];
The stack trace is lying about where the problem occurs - it says that it's in -setUp, but from what I can tell, OCMockito simply reports expectation failures at the beginning of the next unit test's run. The problem location (FooTest.mm:135) looks like the above verify() call.

In fact, all unit tests that fail with this exception fail because we're trying to verify that -[NSObject description] has been called on one mock object or another. Removing all instances of verifying that call makes the tests pass.

I searched on Google for other instances of this particular NSInternalInconsistencyException, and that didn't yield any results. I'm not sure in what way -[NSObject description] is different from any other method that may be called on a mock object - it's possible that the problem isn't really there either, but that it just manifests that way. I was also trying to find if there are any debug/diagnostic options I can turn on for XCTest so I could maybe get more verbose logging information about test execution, but I haven't found anything like that either.

Any ideas for where I should look next? Thanks a lot!

Getting the same error with Xcode's UITesting while running XCTestCase().waitForExpectations — Charlie S, Sep 21 '17 at 09:22
@CharlieSeligman Have you made any progress on that? I tried a couple of things in the meantime - first, I upgraded to the latest OCMockito/OCHamcrest versions, and I'm still getting this error, so it's unlikely that it has to do with old dependencies. Second, using Xcode 8.3.3, I found out that even if I changed the verify() calls to use a wrong invocation count, the tests would still pass, implying that even on Xcode 8 those calls were not working properly, but they did so quietly. — RuslanD, Sep 21 '17 at 22:33
@RusianD - Ended up reworking the method so waitForExpectations was no longer called... so not fixed Im afraid. — Charlie S, Sep 22 '17 at 08:47

Wladek Surala · Accepted Answer · 2017-12-13T17:19:13.837

I experienced a similar problem while working on some big suite of legacy code tests. The question is: are your tests with XCTExpectation working fine when you run them separately? If so, this means that some of the tests that are executed before your tests with NSInternalInconsistencyException are dispatching XCTest related methods in a way which executes them after the relevant test finishes.

it looks like (example):

Test1 -> dispatches asynchronously "block" which executes XCTFail

Test1 -> finishes

XCTFail executed (but Test1 passes, as it finished without "fail") on the main or other thread.

Test2 -> tests something with XCTExpectation -> NSInternalInconsistencyException

Apple docs don't provide much information about internal guts of XCTest, but I'm pretty sure that this is the issue. Try following troubleshooting steps to pin down tests "conflicting" (the bad ones which do asynchronous stuff without XCTestExpectation, like use methods with completion handlers which are eventually doing XCTest assertions, fails etc.):

Do binary search in your suite: disable half of the tests and run the suite with your FooTest.
If your test suite runs fine re-enable half of the disabled tests. If test suite runs with exception, re-enabled all disabled tests and disable other half.
Repeat step 1 then with respect to smaller amount of tests left.
Finally you will end up with tests which cause this exception.

In my case there were multiple tests causing this conflict with XCTestExpectation, hence the search was quite pesky (several hours for a suite of 1000+ XCTestCases, so around 5k tests).

Then investigate thoroughly what happens in the test which is conflicting with your test.

score 0 · Answer 2 · answered Apr 17 '19 at 19:46

It might be the expectation.expectedFulfillmentCount set to a lower number.

expectation.expectedFulfillmentCount = numberOfTimesCalled

Your numberOfTimesCalled could be lower than the number of times the expectation is called. For instance you could be calling twice, but the expectation is set as it would be called once.

Cryptic NSInternalInconsistencyException when running unit tests in Xcode 9 GM

2 Answers2