How to get a vertex position in ARKit's reconstructed mesh?

Question

Recently, I make it about lidar scan project. It is very difficult. And I need to manipulate vertex data. So I tried by this code

guard let meshAnchors = arView.session.currentFrame?.anchors.compactMap { $0 as? ARMeshAnchor } 
else { return }

meshAnchors.first?.geometry.vertices // I want to get vertex position

There is no position of vertex, only buffer data

How can I do that? is it change from buffer data to array? Plz help me.

Answer 1

Just went through this myself so I figured I'd drop ya my solution.

First grab this extension from Apple's Documentation to get a vertex at a specific index:

extension ARMeshGeometry { 
    func vertex(at index: UInt32) -> SIMD3<Float> {
        assert(vertices.format == MTLVertexFormat.float3, "Expected three floats (twelve bytes) per vertex.")
        let vertexPointer = vertices.buffer.contents().advanced(by: vertices.offset + (vertices.stride * Int(index)))
        let vertex = vertexPointer.assumingMemoryBound(to: SIMD3<Float>.self).pointee
        return vertex
    }
}

Then, to get the positions in ARKit world space, you can do something like this:

func getVertexWorldPositions(frame: ARFrame) {
    let anchors = frame.anchors.filter { $0 is ARMeshAnchor } as! [ARMeshAnchor]
    
    // Each mesh geometry lives in its own anchor
    for anchor in anchors {
        
        // Anchor's transform in world space
        let aTrans = SCNMatrix4(anchor.transform)
        
        let meshGeometry = anchor.geometry
        let vertices: ARGeometrySource = meshGeometry.vertices
        
        for vIndex in 0..<vertices.count {
            // This will give you a vertex in local (anchor) space
            let vertex = meshGeometry.vertex(at: UInt32(vIndex))
            // Create a new matrix with the vertex coordinates
            let vTrans = SCNMatrix4MakeTranslation(vertex[0], vertex[1], vertex[2])
            // Multiply it by the anchors's transform to get it into world space
            let wTrans = SCNMatrix4Mult(vTrans, aTrans)
            // Use the coordinates for something!
            let vPos = SCNVector3(wTrans.m41, wTrans.m42, wTrans.m43)
            print(vPos)
        }
    }
}