Looking for an approach to fit rectangles into contourpoints

Question

This is a more general question since I don't know how to tackle or solve this problem efficiently:

• I am using OpenCV to get real-time contour points of moving objects (list of (x,y) coordinate tuples)
• I want to draw rectangles inside these contour points depending if there are angles in it

Suppose the following image shows the contour points of my arm. I want it to detect angles of certain degree and depending on that, draw some rectangles inside.

The whole idea behind this is to make it interact with some game objects (e.g. ball) in pygame later on. For example, move a ball with your hands or a stick or any object that moves in front of the camera.

That is why i want to avoid using more advances libraries like openpose to get the skeleton of my arm since the game should be playable with any kind of object but also run smoothly.

I would be very grateful if you know a suitable approach you can name for this problem!

Approaches I have thought of so far:

• My initial idea is to calculate the distance as well as angle of each neighbor contour points. If the angle is larger than some certain degree it will be considered as a new cluster. However, this doesn't seem to be reliable since the hand (fingers) have sharp edges and i dont want to get a skeleton of small things but rather simple large shapes like in the picture above

• My next idea was to connect all contour points together and form a polygon. However, this would create a complex mask and the contour points are non-constant so it oscillates too much. That is why i thought a simple rectangle should be enough even if it doesn't has a pixel perfect shape

Answer 1

Another Approach is to make a line in between each point and then you can make objects be made out of lines then do line intersection to check for collisions. I recently made a program that does this using math from Wikipedia

#check if 2 lines are intersecting
#lines are 2 pygame Vector2 
def LineIntersect(line1, line2):
    #the math is from wikipedia
    x1 = line1[0].x
    y1 = line1[0].y
    x2 = line1[1].x
    y2 = line1[1].y

    x3 = line2[0].x
    y3 = line2[0].y
    x4 = line2[1].x
    y4 = line2[1].y

    #denominator 
    den = (x1 - x2) * (y3 - y4) - (y1 - y2) * (x3 - x4)
    if den == 0:
        return 
    t = ((x1 - x3) * (y3 - y4) - (y1 - y3) * (x3 - x4)) / den

    u = -((x1 - x2) * (y1 - y3) - (y1 - y2) * (x1 - x3)) / den

    if t > 0 and t < 1 and u > 0 and u < 1:
        pt = Vector2()
        pt.x = x1 + t * (x2 - x1)
        pt.y = y1 + t * (y2 - y1)
        return pt
    return

Another approach that you could do with the first one is to simplify the shape by getting rid of points in a straight line. I did a test and got the following result where the black dots get removed and red is the simplified shape. Not really sure what to do from there, so i guess with my first approach would work best? Here is the code for it

def Prepare(Points):
    New_points = [Points[0]]
    last_point = Vector2(Points[0])
    for i in range(1,len(Points)-1,1):
        p = Vector2(Points[i])
        Dir = p - last_point
        if i < len(Points) - 1:
            New_dir = Points[i+1] - p
            New_dir = New_dir.normalize()
            angle = Dir.angle_to(New_dir)
            if abs(angle) > 15:
                New_points.append(Points[i])
        #print(last_point.angle_to(p))
        pygame.draw.circle(screen,(255,0,0),(int(p.x),int(p.y)),5)    
        last_point = p
    New_points.append(Points[-1])
    return New_points

What i did was got the direction from the previous point to the current point and the next point, if the difference was more than 15 degrees, it was a corner and i added to new points