4

When I went through the video for session 805, Creating Great AR Experiences, and they mention using screen space text when displaying text in AR. They use the example of their Measure app that works quite well.

I want to achieve similar functionality. But, I am not able to figure it out.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
Jagveer Singh
  • 584
  • 7
  • 28

2 Answers2

8

Screen Space is the 2D pixel coordinate system of your display. It's where you normally do things like laying out UIKit widgets in screen space, etc.

The alternative to Screen Space is a 3D coordinate space that relates to the 3D world tracking ARKit does or the 3D rendering engine you use to display virtual 3D content in AR. (In that case there are multiple 3D spaces to consider, like "absolute" world space vs camera-relative positioning.)

As mentioned in the WWDC18 talk Creating Great AR Experiences, screen space is a great way to display content that's inherently 2D (like text) in a way that remains readable regardless of the distance to or orientation of a real-world feature that content is associated with. (If you were to place that text label in world space, it'd get small and unreadable as you get far away.)

ARKit doesn't give you any automatic way to have 2D screen-space content follow a 3D world-space position, but making that happen yourself isn't especially difficult. Here's a general recipe for overlaying UIViews that follow 3D positions:

  1. Add an ARAnchor at the real-world position where you want screen-space content to appear.
  2. Use a per-frame callback (renderer(_:didUpdate:for:) or renderer(_:updateAtTime:) if you're using SceneKit with ARSCNView, session(_:didUpdate:) if you're building your own renderer) to get the current 3D position of that anchor.
  3. Project the 3D position (ARSCNView projectPoint, ARCamera projectPoint) to get the 2D point where that 3D position appears on screen.
  4. Set the position of your UIView to match that point.

For more advanced cases like how the Measure app makes its screen-space label parallel to the line between two 3D points, you'd need to project the two endpoints and use a function like atan2 to get an angle for rotating the label.

rickster
  • 124,678
  • 26
  • 272
  • 326
  • **Project the 3D position (ARSCNView projectPoint, ARCamera projectPoint) to get the 2D point where that 3D position appears on screen.** There is Z also !! . What I need to do with that.and I am also not able to understand XZ plane and XY plane. Detected floor by ARKIt is XZ plane correct !! – Prashant Tukadiya Dec 07 '18 at 06:03
  • One more point is ARSCNView projectPoint give position according to screen orientation. Like If I hold my phone 30 Degree to right side . projection point give rotated shape when use UIBezierPath path. Any help ? – Prashant Tukadiya Dec 07 '18 at 06:29
2

The World (or Absolute) Coordinate System in ARKit is oriented to gravitational force (vertical axis) due to the iPhone's motion sensors by default. The World Coordinates (in 3D space) in ARKit are fixed and a front/rear camera of your iPhone moves within these coordinates. So, if you need to make any manipulations relative to the current position of your iPhone you first need to make a conversion to Camera Space (that also in 3D space).

But Screen Space (a.k.a. Projection Space) is a 2D space of the screen of your iPhone. A small 3D object (for instance 100 meters away from camera) might be seen as a red pixel in Screen Space on your display.

I believe the WWDC'18 presenter tells us about Screen Space in context of robust visibility of text on the screen of your iPhone. That's it.

Read this excellent Matrices post for further details.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220