Cropping/Compositing An Image With Vision/CoreImage

Question

I am working with the Vision framework in iOS 13 and am trying to achieve the following tasks;

Take an image (in this case, a CIImage) and locate all faces in the image using Vision.
Crop each face into its own CIImage (I'll call this a "face image").
Filter each face image using a CoreImage filter, such as a blur or comic book effect.
Composite the face image back over the original image, hereby creating effects that only apply to the face.

A better example of this would be the end goal of taking a live camera feed from an AVCaptureSession and blurring every face in the video frame, compositing the blurred faces back over the original image for saving.

I almost have this working, save for the fact that there seems to be a coordinates/translation issue. For example, when I test this code and move my face, the "blurred" section goes the wrong direction (if I turn my face right, the box goes left, if I look up, the box goes down). While I think this may have something to do with mirroring on the front-facing camera, I can't seem to figure out what I should try next;

func drawFaceBox(bufferImage: CIImage, observations: [VNFaceObservation]) -> CVPixelBuffer? {

    // The filter
    let blur = CIFilter(name: "CICrystallize")

    // The unfiltered image, prepared for filtering
    var filteredImage = bufferImage

    // Find and crop each face
    if !observations.isEmpty {
        for face in observations {
            let faceRect = VNImageRectForNormalizedRect(face.boundingBox, Int(bufferImage.extent.size.width), Int(bufferImage.extent.size.height))
            let croppedFace = bufferImage.cropped(to: faceRect)
            blur?.setValue(croppedFace, forKey: kCIInputImageKey)
            blur?.setValue(10.0, forKey: kCIInputRadiusKey)
            if let blurred = blur?.value(forKey: kCIOutputImageKey) as? CIImage {
                compositorCIFilter?.setValue(blurred, forKey: kCIInputImageKey)
                compositorCIFilter?.setValue(filteredImage, forKey: kCIInputBackgroundImageKey)
                if let output = compositorCIFilter?.value(forKey: kCIOutputImageKey) as? CIImage {
                    filteredImage = output
                }
            }
        }
    }

    // Convert image to CVPixelBuffer and return.  This part works fine.

}

Any thoughts on how I can composite the blurred face image(s) back to their original position with accuracy? Or any other approach to only filter part of the original CIImage to avoid this issue altogether/save processing? Thanks!

Does it work for the back camera? It seems this is an issue of different coordinate systems between Vision and Core Image... — Frank Rupprecht, Jun 05 '20 at 06:00
No, the back camera faces the same issue. I do notice this functions as expected when using the device in “landscape left” orientation, but any other orientation (regardless of camera) is problematic. I’m thinking maybe the orientation is not calculating something when I instantiate the Vision task, but I can’t figure that out. — ZbadhabitZ, Jun 05 '20 at 17:14

score 0 · Answer 1 · answered Jun 06 '20 at 02:19

I believe this issue stems from an orientation problem earlier on in the pipeline (specifically, during the output of the sample buffers from the camera, which is where the Vision task was instantiated). I have updated my didOutputSampleBuffer code like so;

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        ...

        // Setup the current device orientation
        let curDeviceOrientation = UIDevice.current.orientation

        // Handle the image property orientation
        //let orientation = self.exifOrientation(from: curDeviceOrientation)

        // Setup the image request handler
        //let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: CGImagePropertyOrientation(rawValue: UInt32(1))!)
        let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])

        // Setup the completion handler
        let completion: VNRequestCompletionHandler = {request, error in
            let observations = request.results as! [VNFaceObservation]

            // Draw faces
            DispatchQueue.main.async {

                // HANDLE FACES
                self.drawFaceBoxes(for: observations)

            }
        }

        // Setup the image request
        let request = VNDetectFaceRectanglesRequest(completionHandler: completion)

        // Handle the request
        do {
            try handler.perform([request])
        } catch {
            print(error)
        }
    }

As noted, I have commented out the let orientation = ... and the first let handler = ..., which was using the orientation. By removing the reference to the orientation, I seem to have removed any issue with orientation in the Vision calculations.

Cropping/Compositing An Image With Vision/CoreImage

1 Answers1