Calculating angles of body skeleton in video using OpenPose

Question

Disclaimer: This question is regarding OpenPose but the key here is actually to figure how to use the output (coordinates stored in the JSON) and not how to use OpenPose, so please consider reading it to the end.

I have a video of a person from the side on a bike (profile of him sitting so we see the right side). I use the OpenPose to extract the coordinates of the skeleton. The OpenPose provides the coordinates in a JSON file looking like (see docs for explanation):

{
  "version": 1.3,
  "people": [
    {
      "person_id": [
        -1
      ],
      "pose_keypoints_2d": [
        594.071,
        214.017,
        0.917187,
        523.639,
        216.025,
        0.797579,
        519.661,
        212.063,
        0.856948,
        539.251,
        294.394,
        0.873084,
        619.546,
        304.215,
        0.897219,
        531.424,
        221.854,
        0.694434,
        550.986,
        310.036,
        0.787151,
        625.477,
        339.436,
        0.845077,
        423.656,
        319.878,
        0.660646,
        404.111,
        321.807,
        0.650697,
        484.434,
        437.41,
        0.85125,
        404.13,
        556.854,
        0.791542,
        443.261,
        319.801,
        0.601241,
        541.241,
        370.793,
        0.921286,
        502.02,
        494.141,
        0.799306,
        592.138,
        198.429,
        0.943879,
        0,
        0,
        0,
        562.742,
        182.698,
        0.914112,
        0,
        0,
        0,
        537.25,
        504.024,
        0.530087,
        535.323,
        500.073,
        0.526998,
        486.351,
        500.042,
        0.615485,
        449.168,
        594.093,
        0.700363,
        431.482,
        594.156,
        0.693443,
        386.46,
        560.803,
        0.803862
      ],
      "face_keypoints_2d": [],
      "hand_left_keypoints_2d": [],
      "hand_right_keypoints_2d": [],
      "pose_keypoints_3d": [],
      "face_keypoints_3d": [],
      "hand_left_keypoints_3d": [],
      "hand_right_keypoints_3d": []
    }
  ]
}

From what I understand, each JSON is a frame of the video.

My goal is to detect the angles of specific coordinates like right knee, right arm, etc. For example:

openpose_angles = [(9, 10, 11, "right_knee"),
                   (2, 3, 4,   "right_arm")]

This is based on the following OpenPose skeleton dummy:

What I did is to calculate the angle between three coordinates (using Python):

temp_df = json.load(open(os.path.join(jsons_dir, file)))
listPoints = list(zip(*[iter(temp_df['people'][person_number]['pose_keypoints_2d'])] * 3))

count = 0
lmList2 = {}
for x,y,c in listPoints:
    lmList2[count]=(x,y,c)
    count+=1

p1=angle_cords[0]
p2=angle_cords[1]
p3=angle_cords[2]
x1, y1 ,c1= lmList2[p1]
x2, y2, c2 = lmList2[p2]
x3, y3, c3 = lmList2[p3]

# Calculate the angle
angle = math.degrees(math.atan2(y3 - y2, x3 - x2) -
                     math.atan2(y1 - y2, x1 - x2))
if angle < 0:
    angle += 360

This method I saw on some blog (which I forgot where), but was related to OpenCV instead of OpenPose (not sure if makes the difference), but see angles that do not make sense. We showed it to our teach and he suggested us to use vectors to calculate the angles, instead of using math.atan2. But we got confued on how to implment this.

To summarize, here is the question - What will be the best way to calculate the angles? How to calculate them using vectors?

Answer 1

Your teacher is right. I suspect the problem is that 3 points can make up 3 different angles depending on the order. Just consider the angles in a triangle. Also you seem to ignore the 3rd coordinate.

Reconstruct the Skeleton

In your picture you indicate that the edges/bones of the skeleton are

edges = {(0, 1), (0, 15), (0, 16), (1, 2), (1, 5), (1, 8), (2, 3), (3, 4), (5, 6), (6, 7), (8, 9), (8, 12), (9, 10), (10, 11), (11, 22), (11, 24), (12, 13), (13, 14), (14, 19), (14, 21), (15, 17), (16, 18), (19, 20), (22, 23)}

I get the points from your json file with

np.array(pose['people'][0]['pose_keypoints_2d']).reshape(-1,3)

Now I plot that ignoring the 3rd component to get an idea what I am working with. Notice that this does not change the proportions much since the 3rd component is really small compared to the others.

One definitely recognizes an upside down man. I notice that there seems to be some kind of artifact but I suspect this is just an error in recognition and would be better in an other frame.

Calculate the Angle

Recall that the dot product divided by the product of the norm gives the cosine of the angle. See the wikipedia article on dot product. I'll include the relevant picture from that article. So now I can get the angle of two joined edges like this.

def get_angle(edge1,  edge2):
    assert tuple(sorted(edge1)) in edges
    assert tuple(sorted(edge2)) in edges
    edge1 = set(edge1)
    edge2 = set(edge2)
    mid_point = edge1.intersection(edge2).pop()
    a = (edge1-edge2).pop()
    b = (edge2-edge1).pop()
    v1 = points[mid_point]-points[a]
    v2 = points[mid_point]-points[b]

    angle = (math.degrees(np.arccos(np.dot(v1,v2)
                                    /(np.linalg.norm(v1)*np.linalg.norm(v2)))))
    return angle

For example if you wanted the elbow angles you could do

get_angle((3, 4), (2, 3))
get_angle((5, 6), (6, 7))

giving you

110.35748420197164
124.04586139643376

Which to me makes sense when looking at my picture of the skeleton. It's a bit more than a right angle.

What if I had to calculate the angle between two vectors that do not share one point?

In that case you have to be more careful because in that case the vectors orientation matters. Firstly here is the code

def get_oriented_angle(edge1,  edge2):    
    assert tuple(sorted(edge1)) in edges
    assert tuple(sorted(edge2)) in edges
    
    v1 = points[edge1[0]]-points[edge1[1]]
    v2 = points[edge2[0]]-points[edge2[1]]
    angle = (math.degrees(np.arccos(np.dot(v1,v2)   
                              /(np.linalg.norm(v1)*np.linalg.norm(v2)))))
    return angle

As you can see the code is much easier because I don't order the points for you. But it is dangerous since there are two angles between two vectors (if you don't consider their orientation). Make sure both vectors point in the direction of the points you're considering the angle at (both in the opposite direction works too).

Here is the same example as above

get_oriented_angle((3, 4), (2, 3)) -> 69.64251579802836

As you can see this does not agree with get_angle((3, 4), (2, 3))! If you want the same result you have to put the 3 first (or last) in both cases.

If you do

get_oriented_angle((3, 4), (3, 2)) -> 110.35748420197164

It is the same angle as above.

Answer 2

It is probably way too late, but in case some others would like to automatically obtain 2D joint positions, as well as joint and segment angles from a video, I developped a Python package that you can install with pip. https://github.com/davidpagnon/Sports2D/

pip install sports2d

There is also a Colab version that you can use if you want the most user-friendly experience, that installs OpenPose for you and runs on a server.