I'm working on a project using Kinect V2 and pykinect2 to detect gestures and body positions. The issue is that the Kinect's XYZ axis (x=left-right, y=up-down, z=forwards-backwards) are relative to the sensor's pitch, roll, and yaw. This makes it difficult to detect gestures unless the Kinect is positioned exactly as expected. Fortunately, the Kinect API provides a floorPlane with x, y, z, and w values in "Hessian normal form." How can I use these values to represent points in a new coordinate system where the X- and Z-axes are parallel to the floor plane, and the Z-axis is orthogonal to the floor plane? I want the Kinect's body joint coordinates to align with the floor.
I have watched many videos about a bunch of different way to rotate in 3d but I've been nable to take those mathemathical concepts and put them into a reusable function.
I tried to follow this instruction but ultimately failed and I used this quaternion library. I'm using Python 3.7.9 (because pykinect2 is outdated) and the lastest versions of the packages that are still compatible with python 3.7.
This was my latest attempt in doing this. This one just exit the program without any error message (fun)
import numpy as np
import quaternion
from pykinect2 import PyKinectV2, PyKinectRuntime
def get_rotated_joints():
"""Get the rotated joints from the Kinect sensor."""
# Get body frame
if not kinect.has_new_body_frame():
print(f"[get_rotated_joints][{time.strftime('%Y-%m-%d_%H-%M-%S')}]: Waiting for body frame...")
return None
body_frame = kinect.get_last_body_frame()
# Get body
body_frame = kinect.get_last_body_frame()
tracked_bodies = [body for body in body_frame.bodies if body.is_tracked]
if len(tracked_bodies) == 0:
print(f"[get_rotated_joints][{time.strftime('%Y-%m-%d_%H-%M-%S')}]: No bodies tracked...")
return None
body = tracked_bodies[0]
# Define "up" vector
up = np.array([0, 1, 0])
# Get floor plane from Kinect
floor_plane = body_frame.floor_clip_plane
# Extract normal vector from floor plane
floor_normal = np.array([floor_plane.x, floor_plane.y, floor_plane.z])
# Calculate axis and angle of rotation
axis = np.cross(up, floor_normal)
dot = np.dot(up, floor_normal)
angle = np.degrees(np.arccos(dot))
# Define quaternion for rotation
floor_quat = quaternion.from_rotation_vector(axis * angle)
joint_orientations = [[joint_orient.Orientation.x, joint_orient.Orientation.y, joint_orient.Orientation.z] for joint_orient in body.joint_orientations]
# Apply rotation to Kinect orientations
for joint in joint_orientations:
orientation = quaternion.from_float_array(joint)
new_orientation = floor_quat * orientation
new_joint_orientations.append(new_orientation.float_array)
return new_joint_orientations