Swift 4: Vision Camera Preview Issues

Question

For an object recognition app with CoreML and Vision I downloaded a sample app from Apple (https://developer.apple.com/documentation/vision/recognizing_objects_in_live_capture). Unfortunately this app only seems to work in portrait mode but I need to use it landscape.

In standard configuration my objects will get detected but the camera preview layer is rotated 90°. After I rotate the preview layer my objects still get recognized but not as accurate as before. This bugs me and is not acceptable for me nor my client.

I tried to rotate the preview layer like this:

previewLayer.connection?.videoOrientation = .landscapeRight

And tried to rotate the video output like this:

videoDataOutput.connection(with: .video)?.videoOrientation = .landscapeLeft

After the rotation the bounding boxes are not positioned correctly. Moving the camera result in even weirder movements of the boxes. It seems to have something to do with following function:

public func exifOrientationFromDeviceOrientation() -> CGImagePropertyOrientation {

  let curDeviceOrientation = UIDevice.current.orientation
  let exifOrientation: CGImagePropertyOrientation

  switch curDeviceOrientation {
    case UIDeviceOrientation.portraitUpsideDown:  // Device oriented vertically, home button on the top
      exifOrientation = .left

    case UIDeviceOrientation.landscapeLeft:       // Device oriented horizontally, home button on the right
      exifOrientation = .upMirrored

    case UIDeviceOrientation.landscapeRight:      // Device oriented horizontally, home button on the left
      exifOrientation = .down

    case UIDeviceOrientation.portrait:            // Device oriented vertically, home button on the bottom
      exifOrientation = .up

    default:
      exifOrientation = .up
  }

  return exifOrientation
}

If I change

case UIDeviceOrientation.landscapeLeft:
      exifOrientation = .upMirrored

to

case UIDeviceOrientation.landscapeLeft:
      exifOrientation = .left

it will position the bounding boxes correctly and the camera movement seems to work fine as well. But the recognition of the objects suffers big time.

score 0 · Answer 1 · edited Mar 19 '20 at 13:12

0

Using landscape is generally simpler than portrait because the camera's native orientation is already landscape (right). So I'd remove all the rotation / orientation code because that is just confusing things.

In addition, I'd first check what exactly the model is receiving as input, to see that everything looks correct. You can use this model (and tool) for that.

edited Mar 19 '20 at 13:12

Asad Ali Choudhry

4,985
4
31
36

answered Jul 25 '19 at 09:52

Matthijs Hollemans

7,706
2
16
23

I will look into that. Thank you! – kdawg13 Jul 25 '19 at 10:43

score 0 · Answer 2 · answered Jan 03 '20 at 03:29

0

I have changed orientation with

detectionOverlay.setAffineTransform(CGAffineTransform(rotationAngle: CGFloat(.pi / 0.5)).scaledBy(x: scale, y: -scale))

and

captureConnection?.videoOrientation = .landscapeRight

here's a demo I use landscapeRight to detect object. https://github.com/gwh111/CoreMLRecord

answered Jan 03 '20 at 03:29

Gao

85
1
7

Sadly your project doesn't work on iPad with rotation at all – zaitsman Jan 06 '23 at 04:56

Swift 4: Vision Camera Preview Issues

2 Answers2