NSOpenPanel Breaks UI Testing on macOS

Question

I'm using Xcode to do UI testing on a sandboxed macOS app that has the com.apple.security.files.user-selected.read-write entitlement (i.e., can access files and folders explicitly selected by the user via an NSOpenPanel GUI).

I have noticed that code coverage stops right after the open panel is presented modally. This is my code:

@IBAction func go(_ sender: Any) {

    let panel = NSOpenPanel()
    panel.canCreateDirectories = true
    panel.canChooseDirectories = true
    panel.canChooseFiles = false
    panel.allowsMultipleSelection = false

    let response = panel.runModal()

    switch response {
    case NSApplication.ModalResponse.OK:
        openPanelDidSelectURL(panel.urls[0])

    default:
        return
    }
}

(I have recorded my UI tests so that the NSOpenPanel is accepted right away, choosing the folder where it was open.)

Code coverage ends up highlighted like this:

I have tried replacing the switch statement with a fatalError() call, but the UI test still completes successfully, suggesting that anything immediately after:

let response = panel.runModal()

...is not executed during the test.

Disabling sandboxing seems to have no effect, so I suspect it is running the open panel modally that causes trouble...

@Willeke It shouldn't; AppDelegate stores the window controller in a non-weak property; the window controller should live until a new instance is assigned to the property...? Also, the app works as expected when not running the tests (i.e., when debugging). — Nicolas Miari, Mar 20 '19 at 10:04
@Willeke In any case, removing the whole switch clause (`window.close()` included) and replacing it with a call to `fatalError()` does nothing (i.e., the test completes unharmed). Nothing to do with the window closing... — Nicolas Miari, Mar 20 '19 at 10:07
Could you check in the code if it's called on main thread? Model dialogs need this and usually don't enforce in themselves. I also experienced problem with NSOpenPanel in Debug Build when launched from Xcode (see also https://stackoverflow.com/questions/48787777/how-to-acquire-a-file-path-using-objective-c/48894348#48894348) as well as when the App is not properly signed (see https://stackoverflow.com/questions/24692540/nsopenpanel-takes-longer-time-on-mac-osx-10-9/24709660#24709660) — mahal tertin, Mar 20 '19 at 14:33
@mahaltertin My app is signed and sandboxed, but I’ll check about the threads as soon as I can. — Nicolas Miari, Mar 20 '19 at 22:40
@mahaltertinI The open panel is launched from within a button action handler method. I set up a breakpoint during testing and confirmed this method actually does runs on the main thread (as expected). — Nicolas Miari, Mar 22 '19 at 01:04

Nicolas Miari · Accepted Answer · 2019-03-26T01:58:21.753

I tried all other available methods for presenting the open panel, namely:

panel.begin { (response) in
    switch response {
    case NSApplication.ModalResponse.OK:
        self.openPanelDidSelectURL(panel.urls[0])

    default:
        return
    }
}

...and also:

panel.beginSheetModal(for: view.window!) { (response) in
    switch response {
    case NSApplication.ModalResponse.OK:
        self.openPanelDidSelectURL(panel.urls[0])

    default:
        return
    }
}

...but the result is always the same: All code immediately after presenting the panel is not covered during tests.

In the end, I realized that my UI tests cannot rely on some user-selectable folder being present wherever the open panel lands (last visited directory?), so I opted for using mocking instead.

First, in my UI test classes, I adopted this setup logic:

override func setUp() {
    continueAfterFailure = false
    let app = XCUIApplication()
    app.launchArguments.append("-Testing")
    app.launch()
}

(the hyphen before "Testing" is mandatory, otherwise my document-based macOS app will think I am launching it to open a document named "Testing", and fail to do so)

Next, On the app side, I defined a global computed property to determine whether we are running under a test or not:

public var isTesting: Bool {
    return ProcessInfo().arguments.contains("-Testing")
}

Finally, also on the app side I wrapped all NSOpenPanel calls into two methods: One for prompting the user for input files to read, and another to prompt the user for an output directory into which to write the resulting files (this is all my app needs from NSOpenPanel):

public func promptImportInput(completionHandler: @escaping (([URL]) -> Void)) {
    guard isTesting == false else {
        /* 
          Always returns the URLs of the bundled resource files: 
           - 01@2x.png, 
           - 02@2x.png, 
           - 03@2x.png,
             ...
           - 09@2x.png, 
         */
        let urls = (1 ... 9).compactMap { (index) -> URL? in
            let fileName = String(format: "%02d", index) + "@2x"
            return Bundle.main.url(forResource: fileName, withExtension: "png")
        }
        return completionHandler(urls)   
    }
    // (The code below cannot be covered during automated testing)

    let panel = NSOpenPanel()
    panel.canChooseFiles = true
    panel.canChooseDirectories = true
    panel.canCreateDirectories = false
    panel.allowsMultipleSelection = true

    let response = panel.runModal()

    switch response {
    case NSApplication.ModalResponse.OK:
        completionHandler(panel.urls)
    default:
        completionHandler([])
    }
}

public func promptExportDestination(completionHandler: @escaping((URL?) -> Void)) {
    guard isTesting == false else {
        // Testing: write output to the temp directory 
        // (works even on sandboxed apps):
        let tempPath = NSTemporaryDirectory()
        return completionHandler(URL(fileURLWithPath: tempPath))
    }
    // (The code below cannot be covered during automated testing)

    let panel = NSOpenPanel()
    panel.canChooseFiles = false
    panel.canChooseDirectories = true
    panel.canCreateDirectories = true
    panel.allowsMultipleSelection = false

    let response = panel.runModal()

    switch response {
    case NSApplication.ModalResponse.OK:
        completionHandler(panel.urls.first)
    default:
        completionHandler(nil)
    }
}

The portions of these two functions that use the actual NSOpenPanel instead of mocking the user-selected files/directories are still excluded from gathering code coverage statistics (but this time, it's by design).

But at least now it's just this two places. The rest of my code just calls these two functions and does no longer interact with NSOpenPanel directly. I have 'abstracted' the OS's file browsing interface away from my app...

NSOpenPanel Breaks UI Testing on macOS

1 Answers1