Why is Face detection only working with CGImagePropertyOrientation.right?

Question

I've created the Xcode project available on the link below in order to explore the new Vision Framework released by Apple with Xcode9. It is based of adjustments made on codes I've found both on GitHub and Youtube, but I can't understand why the CGImagePropertyOrientation.right on the orientation parameter on the VNImageRequestHandler. Can someone shed a light on this?

GitHub Xcode project

Answer 1

I've found the answer to my own question bu going thru Apple's documentation on UIImage.imageOrientation and CGImagePropertyOrientation. Posting it here in case someone else runs into the same issue: Bottom line is although both of these properties relate to an image orientation, they have different actual values assigned to their enumerations. For example an UIImage.imageOrientation.up will map to value 0 and an CGImagePropertyOrientation.up will map to 1. The way I dealt with it is by building this custom "translation" function that returns the mapping from an UIImage.imageOrientation input to the corresponding CGImagePropertyOrientation value:

func getCGOrientationFromUIImage(_ image: UIImage) -> CGImagePropertyOrientation {
    // returns que equivalent CGImagePropertyOrientation given an UIImage.
    // This is required because UIImage.imageOrientation values don't match to CGImagePropertyOrientation values
    switch image.imageOrientation {
    case .down:
        return .down
    case .left:
        return .left
    case .right:
        return .right
    case .up:
        return .up
    case .downMirrored:
        return .downMirrored
    case .leftMirrored:
        return .leftMirrored
    case .rightMirrored:
        return .rightMirrored
    case .upMirrored:
        return .upMirrored
    }
}

GitHub sample updated. Now faces are recognized no matter which orientation is used for taking the photo.

Answer 2

Face detection requires knowledge of the desired orientation of faces, which typically matches the image's intended display orientation. (That is, most people, most of the time, take pictures where the faces are right side up.) Most face detection software, Vision included, is optimized to look for right-side-up faces, as it's a lot easier to get false positives when you're looking for any-orientation faces.

Vision's face detection features prefer that you specify an orientation. AFAICT they use the default .up orientation if you don't specify one at all. If you have images that recognize correctly only if .right is the specified orientation, those images probably have EXIF metadata indicating that their preferred display orientation is not necessarily the same as the native sensor orientation they were captured with.

If your images are starting as UIImage loaded from storage/URL, you can ask the image what its intended display orientation is with UIImageOrientation, and convert that to CGImagePropertyOrientation for passing to Vision methods. There's a method for doing that in Apple's Vision sample code.