Limiting XCTests measure() to one run only

Question

As part of a UI test I'm writing, I want to measure the performance of a feature. The test itself takes a while, and the part I want to measure is small, and the tests are frequently enough run that I will get the data I need with only one measurement per run. XCTestCase's measure(_:), though, will run it 10 times.

Are there any options I can give it or measureMetrics(_:automaticallyStartMeasuring:for:) to run measurement only once, or is there any way for me to make the measurement myself and feed it back into the test runner so it integrates with Xcode the way measure(_:) does?

Answer 1

This is quite annoying.

Althought in Xcode 11, you can specify XCTMeasureOptions when calling self.measure { }, i.e., measure(options: XCTMeasureOptions, block: () -> Void). In the options, you can set iterationCount as 1.

BUT!

Setting iterationCount as 1, Xcode will actually run the measure block twice, according to the doc of iterationCount:

A performance test runs its block iterationCount+1 times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.

And it did run twice in practice.

Meanwhile setting iterationCount to 0 produces no result cause according to the doc:

... iterationCount+1 times, ignoring the first iteration and recording metrics for the remaining iterations

I don't know if the Runtime Swizzling way of changing the measurement count can get rid of the warm-up run though.

Workaround

Set up a counter to...

1. skip the first run:

    func testYourFunction() {
        let option = XCTMeasureOptions()
        option.iterationCount = 1
        var count = 0
        self.measure(options: option) {
            if count == 0 {
                count += 1
                return
            }
            // Your test code...
        }
    }

2. rollback the change:

    func testYourFunction() {
        let option = XCTMeasureOptions()
        option.iterationCount = 1
        var count = 0
        self.measure(options: option) {
            defer {
                if count == 0 {
                    try? setUpWithError()
                    // call `setUpWithError` manually
                    // or run your custom rollback logic
                }
                count += 1
            }
            // Your test code...
    }