6

Xcode performance tests written with the XCTest.measure() API pass or fail depending on the "baseline" performance which must be set manually in Xcode and stored in ProjectName.xcodeproj/xcshareddata/xcbaselines/. As of Xcode 13.2.1 these baselines can only be set for your current device + simulator combo. For example, if you run the performance tests on another machine with a different processor it will ignore the baseline you set before. This makes sense because different configurations could have very different performance characteristics.

But this presents a problem when running performance test on a headless CI server. How can I set the performance baselines for my headless CI server?

I don't have access to the Xcode UI and xcodebuild doesn't seem to provide a way to set baselines.

And to be clear, the tests do execute sucessfully on my CI server, but they will never fail because baselines are not set.

MattL
  • 1,132
  • 10
  • 23
  • Hey Matt. I'm actually working on it now. Have you found out the way? My current plan: run tests, and copy all artifacts locally, then add new set of data into `.xcodeproj/xcshareddata/xcbaselines` ... so – manual hacking. Stable as long as you don't switch CI runner – Oleg Shanyuk Feb 07 '23 at 14:57
  • Hey @OlegShanyuk, I haven't found any good way to do this still. If you write some custom script to generate the xcbaselines please link it here. – MattL Feb 10 '23 at 12:26
  • – if I'll go down that rabbithole, most likely the final result will be using some sort of Automator... as to the date I did it manually (prev. comment) – and using them now, doing fine-tune by editing plist files in the xcproject/../baselines;... So, first it to ensure it makes sense (and simulator CI data is reliable enough to detect regressions...) – Oleg Shanyuk Feb 11 '23 at 11:09

0 Answers0