Extracting the ARMeshGeometry data to apply AudioKit sound generator

Question

How can I extract and work with the ARMeshGeometry generated by the new SceneReconstruction API on the iPad Pro? I am using Apple's Visualising Scene Semantics sample app/code

I am trying to attach an AudioKit AKOscillator() to the centre of the 'face' as a 3D sound source as its created in realtime.

I can see from the LiDAR example code that this 'seems' to be the point at which a 'face' is created however I am having trouble combining both the extraction/seeing the 'face' data and adding the AudioKit sound source.

Here is where I believe the face is determined (I am new to swift could be very wrong):

DispatchQueue.global().async {
for anchor in meshAnchors {
    for index in 0..<anchor.geometry.faces.count {
        // Get the center of the face so that we can compare it to the given location.
        let geometricCenterOfFace = anchor.geometry.centerOf(faceWithIndex: index)
        
        // Convert the face's center to world coordinates.
        var centerLocalTransform = matrix_identity_float4x4
        centerLocalTransform.columns.3 = SIMD4<Float>(geometricCenterOfFace.0, geometricCenterOfFace.1, geometricCenterOfFace.2, 1)
        let centerWorldPosition = (anchor.transform * centerLocalTransform).position

I would really benefit from seeing the RAW array data if that is achievable? Is this from ARGeometrySource?? Can this be printed or viewed/extracted??

I then want to add something like an oscillator/noise generator to that 'face' in the 3D world location and it be spatialised using the array/location data using something like:

    var oscillator = AKOscillator() Create the sound generator
AudioKit.output = oscillator    Tell AudioKit what to output
AudioKit.start()    Start up AudioKit
oscillator.start()  Start the oscillator
oscillator.frequency = random(in: 220...880)    Set oscillator parameters

I appreciate this is almost a two-fold question however any approach on the ARMeshGemotery data extraction/use or the implementation of a sound source in the centre of each 'face' or both are welcomed. Further code for LiDAR visualising scene semantics example in link above.

Thanks, your assistance is much appreciated,

R

Extracting the ARMeshGeometry data to apply AudioKit sound generator

0 Answers0