Is it possible to send a Selector to the frontmost application's first responder (i.e. the focused element) with permission?

Question

Update

Not sure the accessibility framework is the way to go here. Seems it only lets you interact with other apps, but not do things like programmatically send selectors to them.

For more context, macOS has user-defined key-bindings (~/Library/KeyBindings/DefaultKeyBinding.dict) where you can specify a key binding and a selector that will be sent to whichever app/control has focus when you press it. This happens at the OS level.

Wondering if there's a way to programmatically do something similar, sending selectors from my own app (provided it's requested the appropriate permissions to do so, of course.) I thought that was the Privacy->Accessibility settings, but it looks like I may be going down a wrong path.

Original question.

Just playing around with the accessibility framework, trying to send a selector to the first responder (i.e. the focused element) in the foreground app, regardless if that app is mine or something else.

To test this, I'm trying to send moveToBeginningOfLine: to whichever text field has the focus. If it receives it, the cursor should move to the beginning of the current line.

Now to send it to the first responder in my own application is easy. You use NSApplication.sendAction with a target of 'nil' like in my sendToFirstResponder extension seen below. But this doesn't work with other applications, only your own.

To speak to other applications, you have to use AXUIAutomation (and make sure your app has been granted permissions to use it, which mine has), and even though it appears I am successfully getting a focused AXUIElement in the code below, it never seems to receive the selector as I would expect, and the call reports as failing.

Here's my code I'm playing with...

Selector Extensions:

extension Selector {

    @discardableResult
    func sendToFirstResponder(from sender: Any? = nil) -> Bool {
        return NSApplication.shared.sendAction(self, to: nil, from: sender)
    }

    @discardableResult
    func sendToFrontmostApp() -> Bool {

        let systemWideElement = AXUIElementCreateSystemWide()
        var untypedFocusedElement : AnyObject?

        let getFocusedElementResult = AXUIElementCopyAttributeValue(systemWideElement, kAXFocusedUIElementAttribute as CFString, &untypedFocusedElement)
        guard getFocusedElementResult == .success else {
            print("Couldn't get the focused element. \(getFocusedElementResult)")
            return false
        }

        let focusedElement = untypedFocusedElement as! AXUIElement
        let action = description as CFString

        let performActionResult = AXUIElementPerformAction(focusedElement, action)
        guard performActionResult == .success else {
            print("Couldn't perform the action '\(action)'. \(performActionResult)")
            return false
        }

        return true
    }
}

Test Code:

// If you send this to a TextField, the cursor moves to the beginning of the line
let selectorName = "moveToBeginningOfLine:"
let selector = Selector(selectorName)

// This works if the current app is focused and a TextField is the first responder
let result1 = selector.sendToFirstResponder()
print("Result 1:", result1)

// This always fails on the 'perform action' step, even if there's a focused TextField in the frontmost app
let result2 = selector.sendToFrontmostApp()
print("Result 2:", result2)

Is this possible?

Answer 1

No, you cannot send a selector to an AXUIElement directly in Swift 5. The AXUIElement class in the Accessibility framework provides a way to interact with the accessibility hierarchy of an application, but it doesn't allow you to send arbitrary messages or selectors to a UI element.

However, you can use the AXUIElementPerformAction function to perform certain actions on an AXUIElement. This function takes two parameters: the AXUIElement object to perform the action on, and a constant that specifies the type of action to perform. The available actions are defined in the AXActionConstants enumeration.

Here's an example of how to use AXUIElementPerformAction to perform the "press" action on a button element:

import Cocoa

let buttonElement: AXUIElement = ... // Obtain the AXUIElement for the button

var actionResult: AnyObject?
let action = kAXPressAction as CFString
if AXUIElementPerformAction(buttonElement, action, &actionResult) != .success {
    print("Failed to perform press action on button")
}

In this example, the AXUIElementPerformAction function is called with the buttonElement object and the kAXPressAction constant, which specifies the "press" action to perform. If the action is performed successfully, the actionResult parameter will contain the result of the action (if any).

Note that not all AXUIElements support all actions, so you should check the documentation for the specific element you want to interact with to see what actions are available.