2

I'm using the iOS Vision framework to detect rectangles in real-time with the camera on an iPhone and it works well. The live preview displays a moving yellow rectangle around the detected shape.

However, when the same code is run on an iPad, the yellow rectangle tracks accurately along the X axis, but on the Y it is always slightly offset from the centre and it is not correctly scaled. The included image shows both devices tracking the same test square to better illustrate. In both cases, after I capture the image and plot the rectangle on the full camera frame (1920 x 1080), everything looks fine. It's just the live preview on the iPad that does not track properly.

I believe the issue is caused by how the iPad screen has a 4:3 aspect ratio. The iPhone's full screen preview scales its 1920 x 1080 raw frame down to 414 x 718, where both X and Y dims are scaled down by the same factor (about 2.6). However, the iPad scales the 1920 x 1080 frame down to 810 x 964, which warps the image and causes the error along the Y axis.

A rough solution could be to set a preview layer size smaller than the full screen and have it be scaled down uniformly in a 16:9 ratio matching 1920 x 1080, but I would prefer to use the full screen. Has anyone here come across this issue and found a transform that can properly translate and scale the rect observation onto the iPad screen?

Example test images and code snippet are below.

Comparison of detected rect on iPhone / iPad

let rect: VNRectangleObservation

//Camera preview (live) image dimensions
let previewWidth = self.previewLayer!.bounds.width
let previewHeight = self.previewLayer!.bounds.height

//Dimensions of raw captured frames from the camera (1920 x 1080)
let frameWidth = self.frame!.width
let frameHeight = self.frame!.height

//Transform to change detected rectangle from Vision framework's coordinate system to SwiftUI
let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -(previewHeight))
let scale = CGAffineTransform.identity.scaledBy(x: previewWidth, y: previewHeight)

//Convert the detected rectangle from normalized [0, 1] coordinates with bottom left origin to SwiftUI top left origin
//and scale the normalized rect to preview window dimensions.
var bounds: CGRect = rect.boundingBox.applying(scale).applying(transform)

//Rest of code draws the bounds CGRect in yellow onto the preview window, as shown in the image.
lepapillon
  • 55
  • 5
  • 1
    Well it really depends how are you presenting and scaling the Camera (in UIViewRepreaentable) is it .scaledToFill ? – Mr.SwiftOak Aug 17 '22 at 06:59
  • 1
    Good question -- I hadn't thought of preview layer's properties. Without pasting the entire UIRepresentable, I can confirm I have it set to: `view.videoPreviewLayer.videoGravity = .resizeAspectFill`, and I just changed it to: `view.videoPreviewLayer.videoGravity = .resizeAspect`, and now the rect properly tracks on Y, but not X, and it's framing properly on Y, but stretched out too far on X. I'll look into this some more - thanks for the idea! – lepapillon Aug 17 '22 at 15:12
  • Just to follow up on this: the solution ended up being a combination of changing the preview layer to scale as .resizeAspect, preserving the ratio of the raw frame, and then drawing the rect as a .overlay on the frame view so the rect's coordinates were calculated relative to the origin of the image's aspect-adjusted origin vs. the parent view's origin. This fixed the warping of the rect, and its positioning offset. – lepapillon Aug 17 '22 at 22:16
  • Yeah, glad you foun solution to your problem. You can make it and Answer and accept it. – Mr.SwiftOak Aug 18 '22 at 08:22
  • @lepapillon I'm having trouble drawing the overlay on top of the previewLayer (I'm using an additional CAShapeLayer), can you share the actual drawing code you are using? – cumanzor Aug 21 '22 at 07:44
  • Sure -- I'll edit the answer to add the extra parts in a sec. – lepapillon Aug 23 '22 at 13:19
  • I added the drawing code. Looking at it now, I should clarify that I applied an overlay when drawing the final rectangle over the static image using the .overlay modifier on an Image view, because those were skewed for me. The detected rect corners were 100% accurately shown once I did that. For the live refresh, I added the code to the answer, but I still have a slight warp on the X dim, but it tracks accurately and works for my purposes. I suspect I need to apply a transform on the CGPath to scale it on X by the ratio difference between X and Y. – lepapillon Aug 23 '22 at 13:32

1 Answers1

2

In case it helps anyone else, based on the info posted by Mr.SwiftOak's comment, I was able to resolve the problem through a combination of changing the preview layer to scale as .resizeAspect, rather than .resizeAspectFill, preserving the ratio of the raw frame in the preview. This led to the preview no longer taking up the full iPad screen, but made it a lot simpler to overlay accurately.

I then drew the rectangles as a .overlay to the preview window, so that the drawing coords are relative to the origin of the image (top left) rather than the view itself, which has an origin at (0, 0) top left of the entire screen.

To clarify on how I've been drawing the rects, there are two parts:

  1. Converting the detect rect bounding boxes into paths on CAShapeLayers:
    let boxPath = CGPath(rect: bounds, transform: nil)
        
    let boxShapeLayer = CAShapeLayer()
        
    boxShapeLayer.path = boxPath
    boxShapeLayer.fillColor = UIColor.clear.cgColor
    boxShapeLayer.strokeColor = UIColor.yellow.cgColor
        
    boxLayers.append(boxShapeLayer)
  1. Appending the layers in the updateUIView of the preview UIRepresentable:
    func updateUIView(_ uiView: VideoPreviewView, context: Context)
    {
        if let rectangles = self.viewModel.rectangleDrawings {
            for rect in rectangles {
                uiView.videoPreviewLayer.addSublayer(rect)
            }
        }
    }
zaitsman
  • 8,984
  • 6
  • 47
  • 79
lepapillon
  • 55
  • 5