By default Xcodes performance tests are run ten times and my result is the average of those ten tests. The problem is the averaged result varies considerably each time I run it so I have to run the test at least five times to get a converged result. This is both tedious and time consuming; is there a way to configure either XCode or the unit test itself to run more than ten times?
a class dump of XCTestCase exposes this method:
- (void)_recordValues:(id)arg1 forPerformanceMetricID:(id)arg2 name:(id)arg3 unitsOfMeasurement:(id)arg4 baselineName:(id)arg5 baselineAverage:(id)arg6 maxPercentRegression:(id)arg7 maxPercentRelativeStandardDeviation:(id)arg8 maxRegression:(id)arg9 maxStandardDeviation:(id)arg10 file:(id)arg11 line:(unsigned long long)arg12;
when this method is swizzled the first parameter (arg1) has the 10 durations:
["0.003544568",
"0.003456569",
"0.003198263",
"0.003257955",
"0.003508724",
"0.003454298",
"0.003461192",
"0.00423787",
"0.003359195",
"0.003335757"]
i added 4 new values (1.0, 2.0, 3.0, 4.0) to the end of this list before passing it back to the original implementation, but unfortunately a different class that observes, XCTestLog, has an internal sanity check that gets tripped:
Assertion failure in +[XCTestLog _messageForTest:didMeasureValues:forPerformanceMetricID:name:unitsOfMeasurement:baselineName:baselineAverage:maxPercentRegression:maxPercentRelativeStandardDeviation:maxRegression:maxStandardDeviation:file:line:]
caught "NSInternalInconsistencyException", "Performance Metrics must provide 10 measurements."
once the XCTestLog method is also overridden so it doesn't assert, the additional 4 values can be added without any complaints. unfortunately the view still only shows the 10 results.
it does however update the total time + standard deviation values in the mini view.
Before Swizzling
After Swizzling and adding 4 values
in order to view more than 10 results one would probably have to tweak the XCode runtime to tell the table to show more items.
The short answer: No, there is no current interface exposed to allow a measure block more than ten times.
The longer answer: No, but there is an interface exposed to modify certain metrics of the measure block. The default metrics are in a String array returned from defaultPerformanceMetrics. There appears to only be one metric supported right now: XCTPerformanceMetric_WallClockTime. This only specifies a performance metric that records the time in seconds between calls to a performance test's startMeasuring() and stopMeasuring() methods, and not the number of times the block is run.
In the latest Xcode (11.0+) you don't need swizzling to change iterations count. Use the following function:
func measure(options: XCTMeasureOptions, block: () -> Void)
This will allow you to specify XCTMeasureOptions which has iterationCount property.
Interesting note from docs:
A performance test runs its block
iterationCount+1times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.
I typically expand the Standard Deviation value to include the range I'd accept.
One obvious workaround for you is to perform the operation 5 times within the measure block you are executing. (You'll want to change the expected time.)
measure {
for _ in [1...5] {
// ...
}
}
