Accessibility (Voice Over) with Sprite Kit

Question

I'm attempting to add support for Voice Over accessibility in a puzzle game which has a fixed board. However, I'm having trouble getting UIAccessibilityElements to show up.

Right now I'm overriding accessibilityElementAtIndex, accessibilityElementCount and indexOfAccessibilityElement in my SKScene.

They are returning an array of accessible elements as such:

func loadAccessibleElements()
{
    self.isAccessibilityElement = false

    let pieces = getAllPieces()

    accessibleElements.removeAll(keepCapacity: false)
    for piece in pieces
    {
        let element = UIAccessibilityElement(accessibilityContainer: self.usableView!)

        element.accessibilityFrame = piece.getAccessibilityFrame()
        element.accessibilityLabel = piece.getText()
        element.accessibilityTraits = UIAccessibilityTraitButton
        accessibleElements.append(element)
    }
}

Where piece is a subclass of SKSpriteNode and getAccessibilityFrame is defined:

func getAccessibilityFrame() -> CGRect
{
    return parentView!.convertRect(frame, toView: nil)
}

Right now one (wrongly sized) accessibility element seems to appear on the screen in the wrong place.

Could someone point me in the right direction?

Many thanks

EDIT:
I've tried a hack-ish work around by placing a UIView over the SKView with UIButton elements in the same location as the SKSpriteNodes. However, accessibility still doesn't want to work. The view is loaded as such:

func loadAccessibilityView()
{
    view.isAccessibilityElement = false
    view.accessibilityElementsHidden = false
    skView.accessibilityElementsHidden = false
    let accessibleSubview = UIView(frame: view.frame)
    accessibleSubview.userInteractionEnabled = true
    accessibleSubview.isAccessibilityElement = false
    view.addSubview(accessibleSubview)
    view.bringSubviewToFront(accessibleSubview)

    let pieces = (skView.scene! as! GameScene).getAllPieces()
    for piece in pieces
    {
        let pieceButton = UIButton(frame: piece.getAccessibilityFrame())
        pieceButton.isAccessibilityElement = true
        pieceButton.accessibilityElementsHidden = false
        pieceButton.accessibilityTraits = UIAccessibilityTraitButton
        pieceButton.setTitle(piece.getText(), forState: UIControlState.Normal)
        pieceButton.setBackgroundImage(UIImage(named: "blue-button"), forState: UIControlState.Normal)
        pieceButton.alpha = 0.2
        pieceButton.accessibilityLabel = piece.getText()
        pieceButton.accessibilityFrame = pieceButton.frame
        pieceButton.addTarget(self, action: Selector("didTap:"), forControlEvents: UIControlEvents.TouchUpInside)
        accessibleSubview.addSubview(pieceButton)
    }

    UIAccessibilityPostNotification(UIAccessibilityScreenChangedNotification, nil)

}

The buttons are placed correctly, however accessibility just isn't working at all. Something seems to be preventing it from working.

Further to this, I have tried another way - I added a subview of UIButtons in overlaying the exact same position as the skspritenodes. I was surprised that these buttons too cannot be accessed by voice over. Have I made a mistake blocking inputs? The skscene itself does override touchesBegun(...) etc. — Mason, May 14 '15 at 16:35

score 2 · Accepted Answer · answered May 01 '16 at 21:30

I've searched in vain for a description of how to implement VoiceOver in Swift using SpriteKit, so I finally figured out how to do it. Here's some working code that converts a SKNode to an accessible pushbutton when added to a SKScene class:

// Add the following code to a scene where you want to make the SKNode variable named “leave” an accessible button
// leave must already be initialized and added as a child of the scene, or a child of other SKNodes in the scene
// screenHeight must already be defined as the height of the device screen, in points

// Accessibility

private var accessibleElements: [UIAccessibilityElement] = []

private func nodeToDevicePointsFrame(node: SKNode) -> CGRect {

    // first convert from frame in SKNode to frame in SKScene's coordinates

    var sceneFrame = node.frame
    sceneFrame.origin = node.scene!.convertPoint(node.frame.origin, fromNode: node.parent!)

    // convert frame from SKScene coordinates to device points
    // sprite kit scene origin is in lower left, accessibility device screen origin is at upper left
    // assumes scene is initialized using SKSceneScaleMode.Fill using dimensions same as device points

    var deviceFrame = sceneFrame
    deviceFrame.origin.y = CGFloat(screenHeight-1) - (sceneFrame.origin.y + sceneFrame.size.height)
    return deviceFrame
}

private func initAccessibility() {
    if accessibleElements.count == 0 {
        let accessibleLeave = UIAccessibilityElement(accessibilityContainer: self.view!)
        accessibleLeave.accessibilityFrame = nodeToDevicePointsFrame(leave)
        accessibleLeave.accessibilityTraits = UIAccessibilityTraitButton
        accessibleLeave.accessibilityLabel = “leave” // the accessible name of the button
        accessibleElements.append(accessibleLeave)
    }
}
override func didMoveToView(view: SKView) {
    self.isAccessibilityElement = false
    leave.isAccessibilityElement = true
}

override func willMoveFromView(view: SKView) {
    accessibleElements = []
}

override func accessibilityElementCount() -> Int {
    initAccessibility()
    return accessibleElements.count
}

override func accessibilityElementAtIndex(index: Int) -> AnyObject? {
    initAccessibility()
    if (index < accessibleElements.count) {
        return accessibleElements[index] as AnyObject
    } else {
        return nil
    }
}

override func indexOfAccessibilityElement(element: AnyObject) -> Int {
    initAccessibility()
    return accessibleElements.indexOf(element as! UIAccessibilityElement)!
}

In my case it was enough to override `accessibilityFrame` on node subclass and setup it's label and traits. — Varrry, Nov 25 '18 at 14:15

score 0 · Answer 2 · answered May 14 '15 at 19:37

0

Accessibility frames are defined in the fixed physical screen coordinates, not UIView coordinates, and transforming between them is kind of tricky.

The device origin is the lower left of the screen, with X up, when the device is in landscape right mode.

It's a pain converting, I've no idea why Apple did it that way.

answered May 14 '15 at 19:37

Ed Desert

1
2

Yeah they're a hassle, I've tried a few variations on the getAccessibilityFrame function - all of which didn't work properly. I've managed to get UIButtons overlaid on the sprite elements instead, but accessibility doesn't work for them either. I feel like maybe the SKScene is confusing the inputs? On tap and drag it just makes a noise like you've tried to select an empty area without accessibility. I'll expand on it in my question soon – Mason May 15 '15 at 11:03
I was lucky in that the app spec just wants landscape left&right, so it was easy to hand code the transformation when they didn't consistently work. – Ed Desert May 15 '15 at 15:51
The accessibility element is just a focus area for a tap or gesture, so anything that blocks it would indeed keep it from working. Are you sure your accessibility container, if you have one, is returning "no"? – Ed Desert May 15 '15 at 15:58
In Swift I couldn't seem to override it directly to return false, but I set the variable isAccessibilityElement to false for the UIView associated with the container object. With my new (albeit hackish) UIButton method, I assumed the accessibility would automatically be dealt with like buttons in other view controllers, but something seems amiss. I'll edit my main post to include some of the details – Mason May 15 '15 at 16:59
I never resolved this on iOS6 or iOS7, but the VO focus and switch focus works as expected with iOS 8 and above, i.e. follows the window and not the device orientation. – Ed Desert Jan 25 '17 at 00:19

Accessibility (Voice Over) with Sprite Kit

2 Answers2

Linked