Rendering independent bones

Question

I'm trying to render a model of a hand in a 3D space based on the positions given by the XrHandJointLocationEXT array from the XR_EXT_hand_tracking extension. I am using both GTLF hand models from Valve, which have the correct amount of bones to match the joints defined by the OpenXR specification in the XrHandJointEXT enum.

I do my rendering as follows :

Every frame I update each joint independently by multiplying its current transform with the inverse bind matrice retrieved from the GLTF model. The XrPosef is relative to the center of the 3D space. I am using cglm to handle all the matrice calculations.

for (size_t i = 0; i < XR_HAND_JOINT_COUNT_EXT; ++i) {
    const XrHandJointLocationEXT *joint = &locations->jointLocations[i];
    const XrPosef *pose = &joint->pose;

    glm_mat4_identity(model->bones[i]);

    vec3 position;
    wxrc_xr_vector3f_to_cglm(&pose->position, position);
    glm_translate(model->bones[i], position);

    versor orientation;
    wxrc_xr_quaternion_to_cglm(&pose->orientation, orientation);
    glm_quat_rotate(model->bones[i], orientation, model->bones[i]);

    glm_mat4_mul(model->bones[i], model->inv_bind[i], model->bones[i]);
}

Then the bones array is uploaded to the vertex shader, along with the view-proj matrix, computed for each eye of the HMD.

#version 320 es"

uniform mat4 vp;
uniform mat4 bones[26];

layout (location = 0) in vec3 pos;
layout (location = 1) in vec2 tex_coord;
layout (location = 2) in uvec4 joint;
layout (location = 3) in vec4 weight;

out vec2 vert_tex_coord;

void main() {
    mat4 skin =
        bones[joint.x] * weight.x +
        bones[joint.y] * weight.y +
        bones[joint.z] * weight.z +
        bones[joint.w] * weight.w;
    gl_Position = vp * skin * vec4(pos, 1.0);
    vert_tex_coord = tex_coord;
}

Is the method of calculation correct? I get decent but "glitchy" results. As you can see on the following screenshot, i rendered both independant joints on top of the hand model, and you can see a glitch on the thumb.

Should I take account of the parent bone when computing my bone transform?

The [glTF Skinning Tutorial](https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_020_Skins.md) has some additional math & shader code to review, if you think the error might be there. — emackey, Nov 04 '21 at 12:52

score 0 · Accepted Answer · answered Nov 12 '21 at 19:06

Finally found the answer to my problem: the joint order from the GTLF model doesn't match the order XrHandJointEXT, leading to the right transform being applied to the wrong joint.

In my case, my model defined the Wrist node as being the first one and the Palm as the last one, where OpenXR defines XR_HAND_JOINT_PALM_EXT as the first joint, and XR_HAND_JOINT_WRIST_EXT as the second one.

Here's the code for the update function

bool
wxrc_hand_model_update(struct wxrc_hand_model *model,
                const XrHandJointLocationEXT *locations)
{
        /*
         * OpenXR defines joint 0 as XR_HAND_JOINT_PALM_EXT, but the valve hand
         * model defines Palm as the last joint
         * TODO: handle this dynamically for other models
         */
        static const size_t convert[26] = {
                1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 0
        };

        for (size_t i = 0; i < XR_HAND_JOINT_COUNT_EXT; ++i) {
                const XrHandJointLocationEXT *joint = &locations[convert[i]];
                const XrPosef *pose = &joint->pose;

                mat4 transform = GLM_MAT4_IDENTITY_INIT;

                vec3 position;
                wxrc_xr_vector3f_to_cglm(&pose->position, position);
                glm_translate(transform, position);

                versor orientation;
                wxrc_xr_quaternion_to_cglm(&pose->orientation, orientation);
                glm_quat_rotate(transform, orientation, transform);

                glm_mat4_mul(transform, model->inv_bind[i], model->bones[i]);
        }
        return true;
}

Rendering independent bones

1 Answers1