0

I'm trying to save the camera image from ARFrame to file. The image is given as a CVPixelBuffer. I have the following code, which produces an image with a wrong aspect ratio. I tried different ways, including CIFilter to scale down the image, but still cannot get the correct picture saved.

Camera resolution is 1920w x 1440h. I want to create a 1x scale image from the provided pixel buffer. I am getting 4320 × 5763, 4320 × 3786

How do I save capturedImage (CVPixelBuffer) from ARFrame to file?

func prepareImage(arFrame: ARFrame) {
    
    let orientation = UIInterfaceOrientation.portrait
    let viewportSize = CGSize(width: 428, height: 869)
    
    let transform = arFrame.displayTransform(for: orientation, viewportSize: viewportSize).inverted()
    let ciImage = CIImage(cvPixelBuffer: arFrame.capturedImage).transformed(by: transform)
    let uiImage = UIImage(ciImage: ciImage)
}
Alex Stone
  • 46,408
  • 55
  • 231
  • 407

1 Answers1

1

There are a couple of issues I encountered when trying to pull a screenshot from an ARFrame:

  1. The CVPixelBuffer underlined in the ARFrame is rotated to the left, which means that you will need to rotate the image.
  2. The frame itself is much wider than what you see in the camera. This is done because AR needs a wider range of image to be able to see beyond what you see for it's scene.

E.G: This is an image taken from a raw ARFrame

The following below may assist you to handle those two steps easily: https://github.com/Rightpoint/ARKit-CoreML/blob/master/Library/UIImage%2BUtilities.swift

The problem with these are that this code uses UIGraphicsBeginImageContextWithOptions, which is heavy util.

But if performance are not your main issue(Let's say for an AR Experience) these might be suffice.

If performance is an issue, you can do the mathematical calculation to understand the conversion you will need to do, and simply cut the image needed from the original picture and rotate that.

Hope this will assist you!

YanivH
  • 539
  • 4
  • 18