Xcode 5 says tests failed but shows green checkmarks

Question

I have written a few unit tests in the new XCTest framework of Xcode 5.

The tests seem to pass successfully most of the time, but I've come across a strange behaviour where if I run the tests enough times consecutively, eventually I'll get a 'test failed' message and the console will show Program ended with exit code 0, but there are green checkmarks beside all the tests.

What's going wrong here?

Similar here. For us, it sometimes happens on the first run, then succeeds after that. We also think we fixed it by adding a significant `usleep` into our code, but that is of course unacceptable. (It's also hard to verify that it actually worked.) — 31eee384, Nov 15 '13 at 02:52
Have to say the same thing is happening to me. I have a suite of 72 tests in one project, and only a single test in a second project. Sometimes the notification says it failed although the tests are clearly passing, and sometimes it gives you success. A pattern that gives me a success notification is to run the exact same tests (again I am assuming your tests are passing) twice in a row or three times. Eventually the notification succeeds. — Rui Peres, Oct 14 '13 at 08:22

score 17 · Answer 1 · answered Oct 17 '13 at 14:38

17

This is definitely a bug. Submit a report via https://bugreport.apple.com/ feel free to reference my report (15251050).

answered Oct 17 '13 at 14:38

squarefrog

4,750
4
35
64

Is 15251050 the problem ID? I can't find it. – huggie Oct 30 '13 at 03:31
I can't find it either. – Shinigami Oct 30 '13 at 16:06
That was the reference number Apple gave me. – squarefrog Oct 30 '13 at 16:31
Did your log show your RAM having a ton of memory wired but inactive? I had almost 10GB of 16GB. – GW.Rodriguez Nov 07 '13 at 10:50
Same thing here, but couldn't find the report mentioned. Any ideas? – veducm Nov 20 '13 at 12:04
Apple doesn't let you search for reports, but just mention the report number in the notes. The more duplicated reports they receive, the more likely Apple is to look into it. – squarefrog Nov 21 '13 at 10:53

score 2 · Answer 2 · answered Nov 14 '13 at 03:21

2

Until the bug is fixed, close the project and re-open it.

answered Nov 14 '13 at 03:21

willc2

38,991
25
88
99

score 2 · Answer 3 · answered Jan 28 '14 at 23:08

2

Build Clean worked for me... sometimes anyways. Product -> Clean.

answered Jan 28 '14 at 23:08

ansible

3,569
2
18
29

score 1 · Answer 4 · answered Nov 30 '13 at 23:35

If you look at the output console, you should notice that Xcode starts running test suite:

Test Suite 'SomeClassTests' started at 2013-11-30 23:19:34 +0000

but suddenly exits without running test cases:

Program ended with exit code: 0

and shows a notification that test failed.

Restarting Xcode only helps temporarily as the problem quickly comes back.

I stopped executing tests in Xcode and run all tests by command line using xctool.

RyanM · Answer 5 · 2014-03-03T17:29:01.500

For me, I think this problem was linked to incorrect OCMock configuration. Removing OCMock (completely) and re-adding it to the project exactly as recommended by OCMock seems to have fixed it.

A couple specific things I did wrong:

I was using #import "OCMock.h instead of #import "OCMock/OCMock.h. Xcode autocompleted the former.
I had a bad link leftover in library search paths from previously trying to install OCMock with a different directory structure.
This is dumb, but in trying to debug this issue, I commented out the instantiation of my OCMockObject in my setup method.. and didn't remember to uncomment it. The nil OCMockObject caused similar behavior as described in the other answers here. This is odd.. In my current project I have two tests that use the OCMockObject that is instantiated in my setup. Both should fail. However, one of them passes, and the only difference that I can see between the two is that the one that is passing previously passes legitimately...

Anyways, with OCMock setup correctly my tests are working as they should

Good to know for others, but I did have this problem without OCMock too. — 31eee384, Mar 04 '14 at 21:05

score 0 · Answer 6 · answered Mar 15 '14 at 00:12

Having just run into this in my own updated XCode 5 thought I'd share my similar resolutions.

I notice if I hit any breakpoint when running the test and simply continue it reports as passing, very odd indeed and unacceptable for running your whole suite...

I also noticed adding a usleep as someone mentioned, as well as doing a clean build prior to running the tests seems to resolve the issue. However creating a new test often re-introduces the issue at least for that new test. Hope Apple fixes this soon!

score 0 · Answer 7 · answered Apr 22 '14 at 00:33

0

This appears to have been fixed in XCode 5.1.1

answered Apr 22 '14 at 00:33

Alex

978
7
23

That's what they said, but I could've sworn I saw it again recently. Definitely not as frequently as before, though. – Shinigami May 07 '14 at 14:13
They claim it has been, but the bug remains unfortunately. – Mike May 20 '14 at 14:09
@Shinigami Initially it was working without an issue, but eventually the same thing happened to me. It would be interesting to know what exactly they did that could cause this to happen only less frequently. – Alex May 22 '14 at 16:16

score 0 · Answer 8 · edited Jun 20 '20 at 09:12

0

This bug has been corrected in Xcode 5.1 :

Xcode 5.1 release notes

Notes specifically mention :

Test target [test name] encountered an error (Test process exited with code -1)

Attempt recovery by quitting and restarting the simulator. (15929053)

edited Jun 20 '20 at 09:12

Community

1
1

answered May 13 '14 at 17:46

HackaZach

409
3
15

It was reported as being fixed, but still persists after the update for me. – Mike May 20 '14 at 14:09
It's working out fine for me after the update. Try cleaning the project and restarting computer. – HackaZach May 20 '14 at 17:42
I've done this many times over the last week - nothing so far. – Mike May 20 '14 at 18:25
Sorry to hear that Mike. It must be something else happening in your test setup. This solution worked for me and is the reason why I felt it necessary to contribute. – HackaZach May 22 '14 at 16:54

score 0 · Answer 9 · answered May 29 '14 at 04:15

0

Despite the test I'm using is logic tests, Xcode wants to iPhone simulator to be running. Once in the agony of trying to get tests working again I've reset the iPhone Simulator contents and Settings from it's menu, and this helped.

answered May 29 '14 at 04:15

ReDetection

3,146
2
23
40

Xcode 5 says tests failed but shows green checkmarks

9 Answers9

Linked