failing a XCTestCase with assert without the test continuing to run but without stopping other tests

Question

I'm trying to test my application using the XCTest framework.

I want my single test case to fail if some logical condition holds (using an assertion). I don't want the rest of the code in the test case to run, because this might lead to problems (access to null pointers, for example) I also want the rest of the test case to run normally, and just the failed test to be marked as failed.

I've noticed XCTestCase has a property called continueAfterFailure. However, setting it to YES caused the failed test to continue executing lines after the assertion, and setting it to NO caused the rest of the tests not to run at all.

Is there a solution to this issue?

score 16 · Accepted Answer · answered Nov 19 '14 at 12:25

16

Pascal's answer gave me the idea to achieve this properly. XCTool now behaves like OCUnit when an assertion fails: the execution of the test case is aborted immediately, tearDown invoked and the next test case is run.

Simply override the method invokeTest in your base class (the one that inherits from the XCTestCase class):

- (void)invokeTest
{
    self.continueAfterFailure = NO;

    @try
    {
        [super invokeTest];
    }
    @finally
    {
        self.continueAfterFailure = YES;
    }
}

That's it!

answered Nov 19 '14 at 12:25

Oscar Hierro

1,117
1
10
15

4

Weird but doesn't work in Xcode 9. It still does continue execution after `XCTFail`. – pronebird Jan 28 '18 at 15:57
Even on Xcode 14.0.1 changing self.continueAfterFailure to false on-demand around critical sections of assertions you definitely don't want the test to continue if failures occur (whereas the remainder of the test should have self.continueAfterFailure = true), doesn't seem to always works (almost like the test runner only occasionally checks the flag). The section of assertions I was trying to force self.continueAfterFailure = false around were in an "async" func call so perhaps that is causing some issues? – Bugmeister Oct 21 '22 at 00:24

nCod3d · Answer 2 · 2016-08-29T09:15:09.633

10

The easiest way is to add:

continueAfterFailure = false

into setUp() method. So it will look like this:

Swift

override func setUp() {
    super.setUp()

    continueAfterFailure = false
}

Objective-C

 - (void)setUp {
    [super setUp];

    [self setContinueAfterFailure:NO];
}

edited Aug 29 '16 at 09:15

answered Jun 08 '16 at 13:15

nCod3d

667
1
10
12

2

In the Objective-C, you should be using NO instead of FALSE. – Michael Ozeryansky Aug 26 '16 at 20:56
1

In objC you can use TRUE/FALSE or YES/NO, it doesn't change a thing, it's only a syntactic thing. – nCod3d Aug 28 '16 at 00:37
1

Sure, but you could also say 0 and 1, but we don't since the proper ObjC is YES/NO. Sure it works, go ahead and use it, just know that I won't be the only one to correct it. – Michael Ozeryansky Aug 29 '16 at 01:25
1

You should also set `continueAfterFailure` to `true` inside `tearDown()`, or all testing will stop after a single failure. – Tim Vermeulen Nov 22 '16 at 06:05
4

From [Apple Documentation](https://developer.apple.com/documentation/xctest/xctestcase/1496260-continueafterfailure) : "`continueAfterFailure` - A Boolean value indicating whether the test case should continue running after a failure occurs.". Its related only to single "test case" not all available tests in class. – nCod3d Jul 06 '17 at 10:19

score 7 · Answer 3 · edited Jul 26 '23 at 11:13

7

One option would be to check the condition normally, then fail and return from the test if it is false.

Something like this:

if (!condition) {
  XCTFail(@"o noes");
  return;
}

You could wrap this up in a helper macro to preserve readability.

BDD test libraries like Kiwi are more elegant for this sort of thing, as they make it easier to share setup between many tests which leads to fewer assertions per test.

edited Jul 26 '23 at 11:13

Dominik Palo

2,873
4
29
52

answered Jan 08 '14 at 14:52

Chris Devereux

5,453
1
26
32

+1 I think this is the only way, other than replicating all the `XCTAssertXxx` macros. – trojanfoe Feb 11 '14 at 13:42
1

It's not the only way, check the other answers for a much better approach. – Oscar Hierro Nov 19 '14 at 12:29

score 4 · Answer 4 · answered Apr 29 '14 at 16:18

4

I am able to use continueAfterFailure and let the other tests run by using this pattern:

self.continueAfterFailure = NO;
@try
{
    // Perform test code here
}
@finally
{
    self.continueAfterFailure = YES;
}

answered Apr 29 '14 at 16:18

Pascal Bourque

5,101
2
28
45

1

Thank you for this idea. I've used it to make a better answer which does not require changing the code of the test cases. Check it out. – Oscar Hierro Nov 19 '14 at 12:28

Raphael · Answer 5 · 2018-05-23T07:13:21.117

In Swift projects, I use a helper function (defined in a shared superclass of all my tests which itself extends XCTestCase):

/// Like `XCTFail(...)` but aborts the test.
func XCTAbortTest(_ message: String, 
                  file: StaticString = #file, line: UInt = #line
                 ) -> Never {
    self.continueAfterFailure = false
    XCTFail(message, file: file, line: line)
    fatalError("never reached")
}

As the comment suggests, the call to fatalError is never actually executed; XCTFail aborts the test in an orderly fashion (tearDown is called, next test runs, etc.). The call is only there to trick the compiler into accepting Never as return type since XCTFail returns Void (it does return if continueAfterFailure == true).

Note that self.continueAfterFailure is reset to the default true for every test method. You can also make that explicit in setUp().

I looked into my failures, it looks like sometimes when running tests with xcodebuild, the test doesn't stop on the XCTFail, and will crash the runner on fatalError. I am not able to reproduce this via Xcode though. :( I am setting continueAfterFailure=False — yuf, May 23 '18 at 22:12
@yuf Interesting; I've never tried to run the tests from the CLI. Different behaviour there would probably be a bug? :/ — Raphael, May 24 '18 at 05:18

failing a XCTestCase with assert without the test continuing to run but without stopping other tests

5 Answers5

Linked