How to crop face regions using convexhull polygons

Question

I am using landmark points from dlib library to select the forehead, nose and eye area from my face based on this question: Is there a way to get the area of the forehead (bounding box) by using opencv/dlib and for a live stream video. It works like a charm and i have the points exactly where i want them, what i would like to do is crop the image where the landmarks are set using convexhull polygons.

What i am trying to do is go from this:

to this:

And save it afterwards

Is there a any way to do it? even if it doesn't look pretty. Here's my current code for facial tracking:

import cv2
import dlib
from imutils import face_utils
import imutils
import numpy as np

cap = cv2.VideoCapture(0)


detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_81_face_landmarks.dat")

def face_remap(shape):
    remapped_image = cv2.convexHull(shape)
    return remapped_image


while True:
    _, frame = cap.read()
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    out_face = np.zeros_like(frame)
    faces = detector(gray, 1) 

    for face in faces:
        x1 = face.left()
        y1 = face.top()
        x2 = face.right()
        y2 = face.bottom()
        aam = [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 77, 75, 75, 55, 69, 70, 71, 80, 72, 73, 79, 74, 78]

        landmarks = predictor(gray, face)
        shape = face_utils.shape_to_np(landmarks)
        remapped_shape = np.zeros_like(shape) 
        feature_mask = np.zeros((frame.shape[0], frame.shape[1]))   
        
        x_pts = []
        y_pts = []
        for n in (aam):
            x = landmarks.part(n).x
            y = landmarks.part(n).y

            x_pts.append(x)
            y_pts.append(y)

            cv2.circle(frame, (x, y), 1, (255, 0, 0), -1)
        x1 = min(x_pts)
        x2 = max(x_pts)
        y1 = min(y_pts)
        y2 = max(y_pts)

        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 0, 255), 3)

    cv2.imshow("out", frame)
    k = cv2.waitKey(1)
    if k == 27:
        remapped_shape = face_remap(shape)
        cv2.fillConvexPoly(feature_mask, remapped_shape[0:27], 1)
        feature_mask = feature_mask.astype(np.bool)
        out_face[feature_mask] = frame[feature_mask]
        cv2.imwrite("out_face.png", out_face)
        break
        

    
cap.release()
cv2.destroyAllWindows()

this code will crop the face image from the background and set the landmark as showed in the example. however i would like to crop around the landmarks

the crop you showed is not convex at all. can you be more specific as to what you want? — Shai, Feb 23 '21 at 06:30
Not necessarily needs to use convex hull, the priority is to obtain a cropped image as much similar as described in the example, i thought of convex hull as an example, i'm kinda confused on how it works for specific image shapes — , Feb 24 '21 at 11:10

score 2 · Accepted Answer · answered Feb 23 '21 at 06:43

You selected only a subset of the 81 landmarks dlib identifies on a face, and discarded the landmarks associated with the mouth, chin and the outer contour of the face.

You should do an additional selection leaving only the points at the boundary of the region you are interested in. Furthermore, you should order the selected points such that connecting them, in the right order, will form a polygon marking exactly the region you want to crop.

Once you have the polygon you can use this method to crop it:

Use cv2.fillPoly() to draw mask from the polygon.
Get only the cropped polygon portion of the mask from the image using cv2.bitwsie_and().
Get the bounding rect of the cropped region using cv2.boundingRect().

score 1 · Answer 2 · answered Feb 23 '21 at 06:47

As @Shai mentioned in comments, your shape is not convex at all.

To crop image out of convexhull polygons, you can first get mask out of the convexhull and then apply bitwise and operation on original image using the mask.

Perhaps a function like this would help.

def mask_from_contours(ref_img, contours):
    mask = numpy.zeros(ref_img.shape, numpy.uint8)
    mask = cv2.drawContours(mask, contours, -1, (255,255,255), -1)
    return cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)

If you just want crop the image out of the polygon, you can programmatically connect the landmarks, hard-code by landmark index, and get a mask out of the polygon, and then apply bitwise-and operation.

Here's code sample for filling polygon.

import numpy as np
import cv2
import matplotlib.pyplot as plt

a3 = np.array( [[[10,10],[100,10],[100,100],[10,100]]], dtype=np.int32 )
im = np.zeros([240,320],dtype=np.uint8)
cv2.fillPoly( im, a3, 255 )

How to crop face regions using convexhull polygons

2 Answers2