7

I'm trying to get the bone rotations related to their parents, but I end up getting pretty weird angles.

I've tried everything, matrix multiplications, offsets, axis swapping, and no luck.

guard let bodyAnchor = anchor as? ARBodyAnchor else { continue }

let skeleton = bodyAnchor.skeleton
let jointTransforms = skeleton.jointLocalTransforms

for (i, jointTransform) in jointTransforms.enumerated() {

    //RETRIEVE ANGLES HERE
}

In //RETRIEVE ANGLES HERE I've tried different approaches:

let n = SCNNode()
n.transform = SCNMatrix4(jointTransform)
print(n.eulerAngles)

In this try, I set the jointTransformation to a SCNNode.transform so I can retrieve the eulerAngles to make them human readable and try to understand what's happening.

I get to work some joints, but I think it's pure coincidence or luck, because the rest of the bones rotate very weird.

In other try I get them using jointModelTransforms (Model, instead of Local) so all transforms are relative to the Root bone of the Skeleton.

With this approach I do matrix multiplications like this:

LocalMatrix = Inverse(JointModelMatrix) * (ParentJointModelMatrix)

To get the rotations relative to its parent, but same situation, some bones rotate okay other rotate weird. Pure coincidence I bet.

Why do I want to get the bone rotations?

I'm trying build a MoCap app with my phone that passes to Blender the rotations, trying to build .BVH files from this, so I can use them on Blender.

This is my own rig:

I've done this before with Kinect, but I've been trying for days to do it on ARKit 3 with no luck :(

mnuages
  • 13,049
  • 2
  • 23
  • 40
c4b4d4
  • 964
  • 12
  • 32

2 Answers2

1

Using simd_quatf(from:to:) with the right input should do it. I had trouble with weird angles until i started normalising the vectors:

guard let bodyAnchor = anchor as? ARBodyAnchor else { continue }

let skeleton = bodyAnchor.skeleton
let jointTransforms = skeleton.jointLocalTransforms

for (i, jointTransform) in jointTransforms.enumerated() {
    // First i filter out the root (Hip) joint because it doesn't have a parent
    let parentIndex = skeleton.definition.parentIndices[i]
    guard parentIndex >= 0 else { continue } // root joint has parent index of -1

    //RETRIEVE ANGLES HERE
    let jointVectorFromParent = simd_make_float3(jointTransform.columns.3)
    let referenceVector: SIMD3<Float>
    if skeleton.definition.parentIndices[parentIndex] >= 0 {
         referenceVector = simd_make_float3(jointTransforms[parentIndex].columns.3)
    } else {
         // The parent joint is the Hip joint which should have
         // a vector of 0 going to itself
         // It's impossible to calculate an angle from a vector of length 0,
         // So we're using a vector that's just pointing up
         referenceVector = SIMD3<Float>(x: 0, y: 1, z: 0)
    }
    // Normalizing is important because simd_quatf gives weird results otherwise
    let jointNormalized = normalize(jointVectorFromParent)
    let referenceNormalized = normalize(referenceVector)
    let orientation = simd_quatf(from: referenceNormalized, to: jointNormalized)
    print("angle of joint \(i) = \(orientation.angle)")
}

One important thing to keep in mind though: ARKit3 tracks only some joints (AFAIK the named joints in ARSkeleton.JointName). The other joints are extrapolated from that using a standardized skeleton. Which means, that the angle you get for the elbow for example won't be the exact angle the tracked persons elbow has there.

jaetzold
  • 1,638
  • 14
  • 10
  • Going to try this :-) Will let you know – c4b4d4 Nov 10 '19 at 20:02
  • Tried it, but the angles didn't seem to change. I'm always getting a T-Pose in the `orientation` variable, no matter if person is moving. Am I doing it wrong? I just took the `orientation` variable for each bone and applied it to my rigged character. Should I do something with this orientation before applying it to my character instead? – c4b4d4 Nov 11 '19 at 00:08
  • Ok, TBH what I'm actually using in my own project is the `jointModelTransforms` and then the difference between the joint position and the parent joint position to get the "bone vector". And this definitely works for me. – jaetzold Nov 12 '19 at 10:44
  • Can you elaborate more your answer with code? So I understand what you are doing. I don’t see the jointModelTransform in your posted answer for example. – c4b4d4 Nov 12 '19 at 15:35
0

Just a guess… does this do the job?

  let skeleton = bodyAnchor.skeleton
  let jointTransforms = skeleton.jointLocalTransforms

  for (i, jointTransform) in jointTransforms.enumerated() {
    print(Transform(matrix: jointTransform).rotation)
  }
Tobias Zucali
  • 167
  • 2
  • 9
  • This is what I already tried, and I can tell ArKit 3 is horrible for the moment for MoCap. The T-Pose is not even zero-ed. :( If anyone has achieved anything, leave a comment – c4b4d4 Oct 28 '19 at 01:07