2

I am working on an application with standard AVCaptureDevice flow for getting and displaying frames from iPhone camera, but then it processes them through OpenCV algorithm and puts markers on the display.
The flow is:

  1. Setting up a AVCaptureVideoPreviewLayer.
  2. Getting frames from AVCaptureVideoDataOutputSampleBufferDelegate's function captureOutput
  3. Converting frame to OpenCV mat and processing it.
  4. Getting results from the algorithm as rectangles.
  5. Scaling them back to iPhone's screen and showing in UI.


My current problem is that everything works accurately on rectangle screened devices (iPhone 7, 8, 7 Plus, 8 Plus), but I've got a lot of problems with top-notch devices like iPhone X, iPhone Xs Max and later.
The fact is that due to usage of previewLayer?.videoGravity = .resizeAspectFill on iPhone X family devices the image on the screen (e.g. in AVCaptureVideoPreviewLayer) gets scaled and cropped if to compare with the original from from the camera , but I can't calculate the exact difference to perform correct back scaling.
If I try to render results in OpenCV straight on the device and save them into the memory, the output image is correct. If I do all the scaling and rendering on rectangle-screened devices, the result is also correct. The only problem are top notch devices, as filling their screen with camera frames make them look differently.
I tried getting such methods as metadataOutputRectConverted, but couldn't understand the right usage of the results I get.

let metaRect = self.camera.previewLayer?.metadataOutputRectConverted(fromLayerRect: self.camera.previewLayer?.bounds ?? CGRect.zero) ?? CGRect.zero
// on iPhone 8 I get: (-3.442597823690085e-17, 0.0, 1.0, 1.0)
// so it means that width and height coefficients are 1 and it's nearly not skewed on both x and y,
//so it gives me the right result on the screen

// on iPhone X I get (-3.136083667459394e-17, 0.08949096880131369, 1.0, 0.8210180623973728)
// I see that there's a skew on Y axis and in height, but I don't know how to correctly use it

The code that I use to initialise the layer: 

session.sessionPreset = AVCaptureSession.Preset.hd1920x1080
previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer?.videoGravity = .resizeAspectFill
DispatchQueue.main.async {
            layer.connection?.videoOrientation = orientation
            layer.frame = UIScreen.main.bounds
            view.layer.insertSublayer(layer, at: 0)
}

The code that I use to put objects at screen, resultRect I get from my C++ OpenCV module:

let aspectRatioWidth = CGFloat(1080)/UIScreen.main.bounds.size.width
let aspectRatioHeight = CGFloat(1920)/UIScreen.main.bounds.size.height

let width = CGFloat(resultRect.width) / aspectRatioWidth
let height = CGFloat(resultRect.height) / aspectRatioHeight
let rectx = CGFloat(resultRect.x) / aspectRatioWidth - width / 2.0
let recty = CGFloat(resultRect.y) / aspectRatioHeight - height / 2.0

I would appreciate any help, thank you very much in advance.

inthy
  • 362
  • 6
  • 16

0 Answers0