Spawning model on image tracker with RealityKit

Question

I'm trying to spawn a large 3D model on an image anchor with RealityKit

I want my model to spawn on an image (not track it) and then track the position using world tracking.

This is my code so far, but I'm stuck after watching all youtube tutorials I could find.

import UIKit
import RealityKit

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        let box = try! Experience.loadBox()
        let anchor = AnchorEntity(.image(group: "AR Resources", name: "TrackerImage"))
        anchor.addChild(box)
        
        box.position = anchor.position
        arView.scene.anchors.append(anchor)
    }
}

Hi Simon, RealityKit is a story about tracking. What do you mean saying: `I want my model to spawn on an image (not track it) as to not disappear when the tracking image is outside the camera view`? — Andy Jazz, Mar 09 '22 at 19:13
@AndyJazz Thank you for commenting. Yes, thanks for pointing out, as I'm new to RealityKit. What is the best framework or SDK to use for this purpose? I want my model to spawn where it sees the image, but then track it using world tracking and not image tracking. I've tried following this example aswell, but it seems like the wrong code: https://stackoverflow.com/questions/63078953/can-i-do-arkit-continuous-image-tracking-in-a-world-tracking-configuration-wit I'm also fairly new to working with ios development. — Simon Archer, Mar 10 '22 at 15:03
You can use ARKit+RealityKit or ARKit+SceneKit. For that run `ARWorldTrackingConfig` with `imageDetection` option – https://stackoverflow.com/questions/52898542/which-method-should-be-used-to-achieve-the-most-accurate-world-tracking-experien/52900122#52900122. Also read this – https://stackoverflow.com/questions/62170441/how-to-recognize-a-lot-of-objects-and-add-a-lot-of-nodes-at-the-same-time/62244737#62244737 — Andy Jazz, Mar 10 '22 at 15:13
@AndyJazz Thank you for the links. I will try that. Also learned a lot of new things from reading you posts! — Simon Archer, Mar 10 '22 at 17:59

score 0 · Answer 1 · edited Mar 12 '22 at 11:50

Finally figured out a suitable solution for my situation. For those looking for the same solution, here is my code:

import UIKit
import RealityKit
import ARKit

class ViewController: UIViewController, ARSessionDelegate {
    @IBOutlet var arView: ARView!
    let boxAnchor = try! Experience.loadBox()
    var imageAnchorToEntity: [ARImageAnchor: AnchorEntity] = [:]
    
           
    override func viewDidLoad() {
        super.viewDidLoad()
        boxAnchor.generateCollisionShapes(recursive: true)
        let box = boxAnchor.steelBox as? Entity & HasCollision
        arView.installGestures(for: box!)
        arView.scene.addAnchor(boxAnchor)
        arView.session.delegate = self
    }
           
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        anchors.compactMap { $0 as? ARImageAnchor }.forEach {
            let anchorEntity = AnchorEntity()
            let modelEntity = boxAnchor.steelBox!
                anchorEntity.addChild(modelEntity)
                arView.scene.addAnchor(anchorEntity)
                anchorEntity.transform.matrix = $0.transform
                imageAnchorToEntity[$0] = anchorEntity
            }
        }

        func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
            anchors.compactMap { $0 as? ARImageAnchor }.forEach {
                 let anchorEntity = imageAnchorToEntity[$0]
                     anchorEntity?.transform.matrix = $0.transform
            }
        }
        func installGestures(on object:ModelEntity){
            object.generateCollisionShapes(recursive: true)
            arView.installGestures([.rotation,.scale], for: object)
        }
    }
}

The function;

func installGestures(on object:ModelEntity) {
    object.generateCollisionShapes(recursive: true)
    arView.installGestures([.rotation,.scale], for: object)
}

and this section in the superViewDidLoad()

func installGestures(on object:ModelEntity) {
    object.generateCollisionShapes(recursive: true)
    arView.installGestures([.rotation,.scale], for: object)
}

Is for scaling, rotating and moving the entity, and can be removed if not needed.

The code above was modified from this source: https://developer.apple.com/forums/thread/126845

Spawning model on image tracker with RealityKit

1 Answers1