Is there a way to render Views as Entities in RealityKit?

Question

I've been trying to find a way to display UIViews as 3d objects in my ARView. I have been trying to create a sort-of a box which would act as a display to contain a UIView but with no luck.

This answer provides a solution for SceneKit, but the same does not seem to work in RealityKit. My idea so far was the following, but couldn't find a way to actually hook up a UIView.

let box = MeshResource.generateBox(width: 0.6, height: 0.3, depth: 0.02, cornerRadius: 0.03)
let material = SimpleMaterial()
material.baseColor =  --SET UIVIEW AS DIFFUSE OF MATERIAL?--
let entity  = ModelEntity(mesh: box, materials: [material])
anchorEntity.addChild(entity)

Andy Jazz · Answer 1 · 2020-11-25T15:54:25.490

I see the following solution: You can use snapshotView() or snapshot() instance methods to get a snapshot of a contents of a current UIView (and then you must convert it to .jpg and the use it as a texture for 3D model) or to get a snapshot as .hdr (and then convert it to .jpg as well).

arView.snapshotView(afterScreenUpdates: false)

or:

arView.snapshot(saveToHDR: true, completion: { (image) in ... } )

An approach showing you how to apply a texture on a RealityKit's box primitive:

@IBOutlet var arView: ARView!

override func viewDidLoad() {
    super.viewDidLoad()
    
    arView.environment.background = .color(.systemGray2)
    let scene = try! Experience.loadBox()
    
    var material = SimpleMaterial()
    material.baseColor = try! .texture(.load(named: "snapshot.jpg"))
    
    let mesh = MeshResource.generateBox(size: 0.3)

    let model = ModelEntity(mesh: mesh, materials: [material])
    model.position.x = 0.3
    model.setParent(scene)

    arView.scene.anchors.append(scene)
}

Also, take into consideration that RealityKit 2.0 supports video-textures thru VideoMaterial.

Is there a way to render Views as Entities in RealityKit?

1 Answers1