How to fix IOAF code GPU errors while using ARKit2 & Vision (VNDetectFaceRectanglesRequest) on iPhone XS

Question

While running ARKit on iPhone XS (with iOS 12.1.2 and Xcode 10.1), I'm getting errors and crashes/hangs while running Vision code to detect face bounds.

Errors I'm getting are:

2019-01-04 03:03:03.155867-0800 ARKit Vision Demo[12969:3307770] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2)
2019-01-04 03:03:03.155786-0800 ARKit Vision Demo[12969:3307850] Execution of the command buffer was aborted due to an error during execution. Discarded (victim of GPU error/recovery) (IOAF code 5)
[SceneKit] Error: display link thread seems stuck

This happens on iPhone XS while running the following proof of concept code to reproduce the error (always happens within a few seconds of running the app) - https://github.com/xta/ARKit-Vision-Demo

The relevant ViewController.swift contains the problematic methods:

func classifyCurrentImage() {
    guard let buffer = currentBuffer else { return }

    let image = CIImage(cvPixelBuffer: buffer)
    let options: [VNImageOption: Any] = [:]
    let imageRequestHandler = VNImageRequestHandler(ciImage: image, orientation: self.imageOrientation, options: options)

    do {
        try imageRequestHandler.perform(self.requests)
    } catch {
        print(error)
    }
}

func handleFaces(request: VNRequest, error: Error?) {
    DispatchQueue.main.async {
        guard let results = request.results as? [VNFaceObservation] else { return }
        // TODO - something here with results
        print(results)

        self.currentBuffer = nil
    }
}

What is the correct way to use Apple's ARKit + Vision with VNDetectFaceRectanglesRequest? Getting mysterious IOAF code errors is not correct.

Ideally, I'd also like to use VNTrackObjectRequest & VNSequenceRequestHandler to track requests.

There is decent online documentation for using VNDetectFaceRectanglesRequest with Vision (and without ARKit). Apple has a page here (https://developer.apple.com/documentation/arkit/using_vision_in_real_time_with_arkit) which I've followed, but I'm still getting the errors/crashes.

Answer 1

For anyone else who goes through the pain I just did trying to fix this exact error for VNDetectRectanglesRequest, here was my solution:

It seems that using a CIImage:

let imageRequestHandler = VNImageRequestHandler(ciImage: image, orientation: self.imageOrientation, options: options)

caused Metal to retain a large amount Internal Functions in my memory graph.

I noticed that Apple's example projects all use this instead:

let handler: VNImageRequestHandler! = VNImageRequestHandler(cvPixelBuffer: pixelBuffer,
                                                                    orientation: orientation,
                                                                    options: requestHandlerOptions)

Switching to use cvPixelBuffer instead of a CIImage fixed all of my random GPU timeout errors!

I used these functions to get the orientation (I'm using the back camera. I think you may have to mirror for the front camera depending on what you're trying to do):

func exifOrientationForDeviceOrientation(_ deviceOrientation: UIDeviceOrientation) -> CGImagePropertyOrientation {

    switch deviceOrientation {
    case .portraitUpsideDown:
        return .right

    case .landscapeLeft:
        return .down

    case .landscapeRight:
        return .up

    default:
        return .left
    }
}

func exifOrientationForCurrentDeviceOrientation() -> CGImagePropertyOrientation {
    return exifOrientationForDeviceOrientation(UIDevice.current.orientation)
}

and the following as the options:

var requestHandlerOptions: [VNImageOption: AnyObject] = [:]
let cameraIntrinsicData = CMGetAttachment(pixelBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil)
if cameraIntrinsicData != nil {
            requestHandlerOptions[VNImageOption.cameraIntrinsics] = cameraIntrinsicData
}

Hopefully this saves someone the week that I lost!

Answer 2

You need to call perform method async, just as it is done in the link you've shared. Try below code:

func classifyCurrentImage() {
    guard let buffer = currentBuffer else { return }

    let image = CIImage(cvPixelBuffer: buffer)
    let options: [VNImageOption: Any] = [:]
    let imageRequestHandler = VNImageRequestHandler(ciImage: image, orientation: self.imageOrientation, options: options)

    DispatchQueue.global(qos: .userInteractive).async {
        do {
            try imageRequestHandler.perform(self.requests)
        } catch {
            print(error)
        }
    }
}

Answer 3

Update: from what I can tell, the issue was retain cycles (or the lack of [weak self]) in my demo repo. In Apple's sample project, they properly use [weak self] to avoid retain cycles and ARKit + Vision app runs on the iPhone XS.