7

Using iOS 11 and iOS 12 and ARKit, we are currently able to detect planes on horizontal surfaces, and we may also visualize that plane on the surface.

I am wondering if we can declare, through some sort of image file, specific surfaces in which we want to detect planes? (possibly ignoring all other planes that ARKit detects from other surfaces)

If that is not possible, could we then perhaps capture the plane detected (via an image), to which we could then process through a CoreML model which identifies that specific surface?

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
Michael Ramos
  • 5,523
  • 8
  • 36
  • 51
  • To analyze results detected in the scene, it seems we are given a list of 'ARHitTestResult' objects. But I don't see how we may carve an image out of that. https://developer.apple.com/documentation/arkit/arhittestresult – Michael Ramos Jun 28 '17 at 01:26

2 Answers2

2

ARKit has no support for such thing at the moment. You can indeed capture the plane detected as an image and if you're able to match this through core ML in real time, I'm sure lot of people would be interested!

You should:

  • get the 3D position of the corners of the plane
  • find their 2D position in the frame, using sceneView.projectPoint
  • extract the frame from the currentFrame.capturedImage
  • do an affine transform on the image to be left with the your plane, reprojected to a rectangle
  • do some ML / image processing to detect a match

Keep in mind that the ARKit rectangle detection is often not well aligned, and can have only part of the full plane.

Finally, unfortunately, the feature points that ARKit exposes are not useful since they dont contain any characteristics used for matching feature points across frames, and Apple has not say what algorithm they use to compute their feature points.

Guig
  • 9,891
  • 7
  • 64
  • 126
  • I think apple has a demo app with qr code extraction where the do the affine transformation form the 2D corners, but I could not remember where I saw it. – Guig Jun 30 '17 at 19:16
  • @Guig sceneView.projectPoint for node positions on plane give irregular shapes for projected view coordinates. Any help? – Roshan Bade Aug 21 '19 at 05:56
-1

Here is small demo code for Find horizontal surface. In #Swift5 Github

Arjun Patel
  • 1,394
  • 14
  • 23