How to detect a 2D image using ARKit and RealityKit?

Question

I want to detect a 2D image using ARKit and RealityKit. I don't want to use SceneKit since of many implementations based on RealityKit (access to Experience.rcproject). Any ideas?

I've tried this code

import ARKit
import RealityKit


class ViewController: UIViewController, ARSessionDelegate {

    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {

        guard let imageAnchor = anchors.first as? ARImageAnchor,
              let _ = imageAnchor.referenceImage.name
        else { return }

        let anchor = AnchorEntity(anchor: imageAnchor)

        // Add Model Entity to anchor
        anchor.addChild(model)

        arView.scene.anchors.append(anchor)
    }

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        arView.session.delegate = self
        resetTrackingConfig()
    }

    func resetTrackingConfig() {

        guard let refImg = ARReferenceImage.referenceImages(inGroupNamed: "Sub",
                                                                  bundle: nil)
        else { return }

        let config = ARWorldTrackingConfiguration()
        config.detectionImages = refImg
        config.maximumNumberOfTrackedImages = 1

        let options = [ARSession.RunOptions.removeExistingAnchors,
                       ARSession.RunOptions.resetTracking]

        arView.session.run(config, options: ARSession.RunOptions(options))
    }
}

From a previous post but it seems to be outdated/incomplete. The code below works for SceneKit but I can't access any scenes from Experience.rcproject

import UIKit
import RealityKit
import SceneKit
import ARKit

class ViewController: UIViewController, ARSCNViewDelegate {

    @IBOutlet var sceneView: ARSCNView!

    override func viewDidLoad() {
        super.viewDidLoad()
        sceneView.delegate = self
    }

    override func viewWillAppear(_ animated: Bool) {
            super.viewWillAppear(animated)

            let configuration = ARImageTrackingConfiguration()

            guard let trackedImages = ARReferenceImage.referenceImages(inGroupNamed: "Photos", bundle: Bundle.main) else {
                print("No images available")
                return
            }

            configuration.trackingImages = trackedImages
            configuration.maximumNumberOfTrackedImages = 7

            sceneView.session.run(configuration)
        }

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
        <#code#>
    }

        func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {

            let node = SCNNode()

            if let imageAnchor = anchor as? ARImageAnchor {
                let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)

                plane.firstMaterial?.diffuse.contents = UIColor(white: 1, alpha: 0.8)

                let planeNode = SCNNode(geometry: plane)
                planeNode.eulerAngles.x = -.pi / 2

                let shipScene = SCNScene(named: "ship.scn")!
                let shipNode = shipScene.rootNode.childNodes.first!

                shipNode.position = SCNVector3Zero
                shipNode.position.z = 0.15

                planeNode.addChildNode(shipNode)

                node.addChildNode(planeNode)
            }
            return node
        }
}

score 1 · Answer 1 · answered Jul 24 '20 at 12:48

As an alternative to including the reference image(s) in your bundle, add your image(s) - at runtime if you want to - like so:

a. Set up a dictionary to help associate your images with the anchors:

private var imageAnchorToEntity: [ARImageAnchor: AnchorEntity] = [:]

b. Set your reference images in viewDidLoad() or at any other location of your choice:

let exampleImage = ARReferenceImage(UIImage(named: "example"), orientation: .up, physicalWidth: 0.2)
let set = Set<ARReferenceImage>()
set.insert(exampleImage) 
let config = ARImageTrackingConfiguration()
config.trackingImages = set

c. In the delegate method that is called after an ARAnchor is added by ARKit, identify the anchor as an ARImageAnchor, create an AnchorEntity from it and associate it with the ARImageAnchor for future reference using the dictionary:

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    anchors.compactMap { $0 as? ARImageAnchor }.forEach {
        let anchorEntity = AnchorEntity(anchor: $0)
        // Add other entities as children to your AnchorEntity
        anchorEntity.addChild(yourEntity)
        imageAnchorToEntity[$0] = anchorEntity // Setting the reference
        arView.scene.addAnchor(anchorEntity)
}

d. Accordingly, you can get a reference to your AnchorEntity and its children in the session(_:didUpdate:) delegate method by accessing your dictionary:

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    anchors.compactMap { $0 as? ARImageAnchor }.forEach {
        let anchorEntity = imageAnchorToEntity[$0]
        anchorEntity?.transform.matrix = $0.transform
    }
}

Note: I've used an ARImageTrackingConfiguration in this case, which is used primarily for tracking images. It works on a ARWorldTrackingConfiguration as well, but ARKit might not be able to track your images as well as the user moves the device.

score 0 · Answer 2 · answered Jun 06 '20 at 20:19

Add a new "AR Resource Group" to your project assets and add your marker image there. Make sure to give it a width and height. I called the group "Markers"
Add the following code (I have it in the viewWillAppear) of my UIViewController:

    let imageAnchor = AnchorEntity(.image(group: "Markers", name: "YourImageName"))
    yourARView.scene.anchors.append(imageAnchor)
    // Setup some models to add etc
    imageAnchor.addChild(someModel)

How to detect a 2D image using ARKit and RealityKit?

2 Answers2