7

Disclaimer: This question is regarding OpenPose but the key here is actually to figure how to use the output (coordinates stored in the JSON) and not how to use OpenPose, so please consider reading it to the end.

I have a video of a person from the side on a bike (profile of him sitting so we see the right side). I use the OpenPose to extract the coordinates of the skeleton. The OpenPose provides the coordinates in a JSON file looking like (see docs for explanation):

{
  "version": 1.3,
  "people": [
    {
      "person_id": [
        -1
      ],
      "pose_keypoints_2d": [
        594.071,
        214.017,
        0.917187,
        523.639,
        216.025,
        0.797579,
        519.661,
        212.063,
        0.856948,
        539.251,
        294.394,
        0.873084,
        619.546,
        304.215,
        0.897219,
        531.424,
        221.854,
        0.694434,
        550.986,
        310.036,
        0.787151,
        625.477,
        339.436,
        0.845077,
        423.656,
        319.878,
        0.660646,
        404.111,
        321.807,
        0.650697,
        484.434,
        437.41,
        0.85125,
        404.13,
        556.854,
        0.791542,
        443.261,
        319.801,
        0.601241,
        541.241,
        370.793,
        0.921286,
        502.02,
        494.141,
        0.799306,
        592.138,
        198.429,
        0.943879,
        0,
        0,
        0,
        562.742,
        182.698,
        0.914112,
        0,
        0,
        0,
        537.25,
        504.024,
        0.530087,
        535.323,
        500.073,
        0.526998,
        486.351,
        500.042,
        0.615485,
        449.168,
        594.093,
        0.700363,
        431.482,
        594.156,
        0.693443,
        386.46,
        560.803,
        0.803862
      ],
      "face_keypoints_2d": [],
      "hand_left_keypoints_2d": [],
      "hand_right_keypoints_2d": [],
      "pose_keypoints_3d": [],
      "face_keypoints_3d": [],
      "hand_left_keypoints_3d": [],
      "hand_right_keypoints_3d": []
    }
  ]
}

From what I understand, each JSON is a frame of the video.

My goal is to detect the angles of specific coordinates like right knee, right arm, etc. For example:

openpose_angles = [(9, 10, 11, "right_knee"),
                   (2, 3, 4,   "right_arm")]

This is based on the following OpenPose skeleton dummy:

enter image description here

What I did is to calculate the angle between three coordinates (using Python):

temp_df = json.load(open(os.path.join(jsons_dir, file)))
listPoints = list(zip(*[iter(temp_df['people'][person_number]['pose_keypoints_2d'])] * 3))

count = 0
lmList2 = {}
for x,y,c in listPoints:
    lmList2[count]=(x,y,c)
    count+=1

p1=angle_cords[0]
p2=angle_cords[1]
p3=angle_cords[2]
x1, y1 ,c1= lmList2[p1]
x2, y2, c2 = lmList2[p2]
x3, y3, c3 = lmList2[p3]

# Calculate the angle
angle = math.degrees(math.atan2(y3 - y2, x3 - x2) -
                     math.atan2(y1 - y2, x1 - x2))
if angle < 0:
    angle += 360

This method I saw on some blog (which I forgot where), but was related to OpenCV instead of OpenPose (not sure if makes the difference), but see angles that do not make sense. We showed it to our teach and he suggested us to use vectors to calculate the angles, instead of using math.atan2. But we got confued on how to implment this.

To summarize, here is the question - What will be the best way to calculate the angles? How to calculate them using vectors?

vesii
  • 2,760
  • 4
  • 25
  • 71
  • 2
    `atan2` is really good to calculate angles, and "use vectors instead" is not a particularly helpful advice since `math.atan2(y3 - y2, x3 - x2) - math.atan2(y1 - y2, x1 - x2)` is quite litterally calculating the oriented angle between the two vectors ((x2, y2), (x1, y1)) and ((x2, y2), (x3, y3)). – Stef Sep 12 '21 at 20:53
  • However, I have no idea what the list of numbers in `"pose_keypoints_2d"` in your json file represents? It's just a list of numbers. How can you identify the (x, y) coordinates of the right knee, and the (x, y) coordinates of the right arm, etc, in that list? – Stef Sep 12 '21 at 20:54
  • All the magic is apparently happening in `listPoints = list(zip(*[iter(temp_df['people'][person_number]['pose_keypoints_2d'])] * 3))`. Do you understand that line of code and what it does? – Stef Sep 12 '21 at 20:57
  • Related question: [How does `zip(*[iter(s)]*n)` work in python?](https://stackoverflow.com/questions/2233204/how-does-zipitersn-work-in-python) – Stef Sep 12 '21 at 20:58
  • You can't calculate like this. **I'm sure you don't know the joint range of the man on the bike.** So not magic function for you ! – dsgdfg Oct 04 '21 at 06:30

2 Answers2

3

Your teacher is right. I suspect the problem is that 3 points can make up 3 different angles depending on the order. Just consider the angles in a triangle. Also you seem to ignore the 3rd coordinate.

Reconstruct the Skeleton

In your picture you indicate that the edges/bones of the skeleton are

edges = {(0, 1), (0, 15), (0, 16), (1, 2), (1, 5), (1, 8), (2, 3), (3, 4), (5, 6), (6, 7), (8, 9), (8, 12), (9, 10), (10, 11), (11, 22), (11, 24), (12, 13), (13, 14), (14, 19), (14, 21), (15, 17), (16, 18), (19, 20), (22, 23)}

I get the points from your json file with

np.array(pose['people'][0]['pose_keypoints_2d']).reshape(-1,3)

Now I plot that ignoring the 3rd component to get an idea what I am working with. Notice that this does not change the proportions much since the 3rd component is really small compared to the others. skeleton

One definitely recognizes an upside down man. I notice that there seems to be some kind of artifact but I suspect this is just an error in recognition and would be better in an other frame.

Calculate the Angle

Recall that the dot product divided by the product of the norm gives the cosine of the angle. See the wikipedia article on dot product. I'll include the relevant picture from that article. Angle from dot product So now I can get the angle of two joined edges like this.

def get_angle(edge1,  edge2):
    assert tuple(sorted(edge1)) in edges
    assert tuple(sorted(edge2)) in edges
    edge1 = set(edge1)
    edge2 = set(edge2)
    mid_point = edge1.intersection(edge2).pop()
    a = (edge1-edge2).pop()
    b = (edge2-edge1).pop()
    v1 = points[mid_point]-points[a]
    v2 = points[mid_point]-points[b]

    angle = (math.degrees(np.arccos(np.dot(v1,v2)
                                    /(np.linalg.norm(v1)*np.linalg.norm(v2)))))
    return angle

For example if you wanted the elbow angles you could do

get_angle((3, 4), (2, 3))
get_angle((5, 6), (6, 7))

giving you

110.35748420197164
124.04586139643376

Which to me makes sense when looking at my picture of the skeleton. It's a bit more than a right angle.

What if I had to calculate the angle between two vectors that do not share one point?

In that case you have to be more careful because in that case the vectors orientation matters. Firstly here is the code

def get_oriented_angle(edge1,  edge2):    
    assert tuple(sorted(edge1)) in edges
    assert tuple(sorted(edge2)) in edges
    
    v1 = points[edge1[0]]-points[edge1[1]]
    v2 = points[edge2[0]]-points[edge2[1]]
    angle = (math.degrees(np.arccos(np.dot(v1,v2)   
                              /(np.linalg.norm(v1)*np.linalg.norm(v2)))))
    return angle

As you can see the code is much easier because I don't order the points for you. But it is dangerous since there are two angles between two vectors (if you don't consider their orientation). Make sure both vectors point in the direction of the points you're considering the angle at (both in the opposite direction works too).

Here is the same example as above

get_oriented_angle((3, 4), (2, 3)) -> 69.64251579802836

As you can see this does not agree with get_angle((3, 4), (2, 3))! If you want the same result you have to put the 3 first (or last) in both cases.

If you do

get_oriented_angle((3, 4), (3, 2)) -> 110.35748420197164

It is the same angle as above.

Lukas S
  • 3,212
  • 2
  • 13
  • 25
  • 1
    Friend, this an **virtual calculation**, don't forget `human is real`. – dsgdfg Oct 04 '21 at 06:31
  • @dsgdfg Hi, thanks for your comment but can you explain this in a few more words? I don't understand what you wanted to say. – Lukas S Oct 04 '21 at 10:16
  • Is the artifact you're describing the value(s) at the origin? Looking at the original dataset, nodes 16 and 18 appear to be (0,0,0) suggesting to me that they're probably not present. I think you could safely just remove those two nodes. Also, I think the algorithm in your angle function is quite elegant, although I don't know about the practice of giving the variable in the function the same as the function. Is that good practice? Anyway, the key point is the ordering of the points; I doubt using cos vs atan2 should be an issue. – ramzeek Oct 05 '21 at 03:38
  • @wikikikitiki Yea I agree that 16 and 18 being (0,0,0) probably means that they are not present. I think it's better to write a comment rather than removing them though. The latter potentially confuses the op and does not help with the question. – Lukas S Oct 05 '21 at 04:00
  • @wikikikitiki Haha I hadn't noticed that I called both things angle. Seems alrightish but I changed it. That's more clean. – Lukas S Oct 05 '21 at 04:00
  • 1
    @wikikikitiki I must disagree about the atan though. The op had ignored the 3rd component making it a 2d calculation. I don't think that the atan calculation is as straight forward as the one with cos in 3d. I am open to be enlightened though. – Lukas S Oct 05 '21 at 04:03
  • @user2640045, thinking that we're in 3D, you're right. To save face, I'll point out that the third dimension here makes very little difference, but I know it's a losing argument. :) – ramzeek Oct 05 '21 at 04:15
  • @wikikikitiki Ah that's reassuring. No going back to school for me few. Have a nice day. – Lukas S Oct 05 '21 at 04:34
  • @user2640045 The effect of angular deviations on the vector, converting coordinate points to closed loops (ABS-INC). Calculation of isometric (non-standard) angles of objects, angle and time matching and control of object dimensions. Your answer does not include these important points. – dsgdfg Oct 06 '21 at 09:21
  • 1
    @dsgdfg All these points don't seem to be about the thing the op asked about "This question is regarding [...] how to use the output [...] and not how to use OpenPose." I maybe wrong about that. If so please feel free to add you own answer addressing one or all your points or point out one or more specific things in the comments. I just tried to answer the question at hand to the best of my ability. – Lukas S Oct 08 '21 at 16:13
  • @user2640045 Thanks for the all information (sorry that could not check it for a while). I have only one question regarding the `get_angle` method. I see that this method assumes that the two vectors share one point (the mid point). What if I had to calculate the angle between the back the the right leg (for the side it makes sense) so coords `(1,8)` and `(9,10)`. How can we calculate the angle in that case? – vesii Oct 16 '21 at 11:37
  • @vesii I have added a paragraph about it. – Lukas S Oct 16 '21 at 13:49
  • @user2640045 thank you very much for the additional paragraph. So how do I consider the orientation of the angles? In other words, how do I sort them right? If I'm looking at the person from the side, why `get_oriented_angle((3, 4), (3, 2))` is the correct one? Does it matter if `get_oriented_angle((3, 2), (3, 4))`? – vesii Oct 19 '21 at 12:52
  • @vesii no the order of the vectors does not matter. `get_oriented_angle((3, 4), (3, 2))== get_oriented_angle((3, 2), (3, 4)).` Only the order of the points. Imagine it to be two crossing lines. They make up two angles. One being 180° minus the other. If you swap the order of one of the point pairs. You get the other angle. So `get_oriented_angle((3, 2), (3, 4))` is the same as `180-get_oriented_angle((2, 3), (3, 4))`. Notice I swapped the 3 and the 2. – Lukas S Oct 19 '21 at 15:09
  • @vesii Was that understandable? Are you happy now? – Lukas S Oct 23 '21 at 19:30
0

It is probably way too late, but in case some others would like to automatically obtain 2D joint positions, as well as joint and segment angles from a video, I developped a Python package that you can install with pip. https://github.com/davidpagnon/Sports2D/

pip install sports2d

There is also a Colab version that you can use if you want the most user-friendly experience, that installs OpenPose for you and runs on a server.