Is there a way to run a test using XCTest multiple times just with different values?

Question

I'm currently working on a project trying to determine how long different sorting algorithms take to sort different sized arrays. To measure the time, I've decided to use XCTest in Swift Playgrounds since it can automate the process of running the algorithm multiple times and averaging it out. But I have an issue with this method because I have to test a large variety array sizes from 15 elements up to 1500 or so, at 5 element intervals (ie. 15 elements, 20 elements, 25 elements...).

The only way I've been able to do this with one test is multiple functions with the different size and measuring the performance. Here is a sample of what that looks like:

class insertionSortPerformanceTest: XCTestCase {
    func testMeasure10() {
        measure {
            _ = insertionSort(arrLen: 10)
        }
    }
    func testMeasure15() {
        measure {
            _ = insertionSort(arrLen: 15)
        }
    }
    func testMeasure20() {
        measure {
            _ = insertionSort(arrLen: 20)
        }
    }
}

insertionSort() works by generating an array of length arrLen and populating it with random numbers.

Is there a way to automate this process somehow?

Also, is there a way to take the output in the console and save it as a string so I can parse it for the relevant information later?

I can't help with the latter portion, but you *can't* test performance in a playgrounds. — Alexander, Oct 21 '20 at 02:58
@Alexander-ReinstateMonica Are you sure? Because I am able to get the output of `measure` when I run it in my playground. — wrongsyntax, Oct 21 '20 at 03:14
Sure, but it's the runtime of un-optimized code, which is **completely** meaningless. Debug builds optimize for fast compilation. Playgrounds further slow down performance by adding in all kinds of hooks to update the UI (to populate the previews on the sidebar). You need an optimized release build, which takes longer to compiler, but whose measurements actually might mean something. E.g. `forEach` calls might run slower than `for` loops in a debug build, because they might not get inlined (in an optimized build, they produce the same code) — Alexander, Oct 21 '20 at 13:02
@Alexander-ReinstateMonica Do you know how I could achieve that? — wrongsyntax, Oct 22 '20 at 00:42
Check out https://forums.swift.org/t/how-do-i-set-up-a-performance-test-suite/36421 — Alexander, Oct 22 '20 at 01:27

score 0 · Answer 1 · answered Sep 02 '23 at 12:59

XCTest doesn't have built-in support for parameterized tests in the way that some other testing frameworks do, but you can simulate this feature with loops within your test functions.

Here is an example that iterates through an array of array lengths and calls insertionSort(arrLen:) for each one:

class insertionSortPerformanceTest: XCTestCase {
    func testInsertionSortPerformance() {
        let arrayLengths = Array(stride(from: 15, through: 1500, by: 5))
        
        for arrLen in arrayLengths {
            measureMetrics([.wallClockTime], automaticallyStartMeasuring: false) {
                startMeasuring()
                _ = insertionSort(arrLen: arrLen)
                stopMeasuring()
            }
        }
    }
}

The use of automaticallyStartMeasuring: false allows you to manually control the measuring process. You start measuring with startMeasuring() and stop it with stopMeasuring().

Capture Console Output Capturing the console output is a bit trickier because XCTest itself doesn't provide a direct way to save test logs to a string. However, there are ways to capture stdout and stderr in Swift that you can use to collect these logs.

Here's an example function to capture the standard output temporarily:

func captureStandardOutput(closure: () -> ()) -> String {
    let originalStdout = dup(STDOUT_FILENO)
    let pipe = Pipe()
    dup2(pipe.fileHandleForWriting.fileDescriptor, STDOUT_FILENO)

    closure()

    let data = pipe.fileHandleForReading.readDataToEndOfFile()
    if let output = String(data: data, encoding: .utf8) {
        dup2(originalStdout, STDOUT_FILENO)
        close(originalStdout)
        return output
    }
    return ""
}

You can use this function like this:

class insertionSortPerformanceTest: XCTestCase {
    func testInsertionSortPerformance() {
        // Your code
        let output = captureStandardOutput {
            // Your test logic here
        }
        print("Captured output: \(output)")
    }
}

Just be cautious when using this approach, as it involves changing the file descriptors for stdout, which can be risky in a multi-threaded environment. Always restore the original stdout file descriptor after capturing the output to avoid affecting other parts of your application.

You can then parse the output string to extract the information you want to collect.

Is there a way to run a test using XCTest multiple times just with different values?

1 Answers1