Cropping face using dlib facial landmarks

Question

I am trying to crop a face using the facial landmarks identified by dlib. The right eyebrow is causing problems - the crop goes flat across rather than follow the eyebrow arc.

What am I doing wrong here?

from imutils import face_utils
import imutils
import numpy as np
import collections
import dlib
import cv2

def face_remap(shape):
   remapped_image = shape.copy()
   # left eye brow
   remapped_image[17] = shape[26]
   remapped_image[18] = shape[25]
   remapped_image[19] = shape[24]
   remapped_image[20] = shape[23]
   remapped_image[21] = shape[22]
   # right eye brow
   remapped_image[22] = shape[21]
   remapped_image[23] = shape[20]
   remapped_image[24] = shape[19]
   remapped_image[25] = shape[18]
   remapped_image[26] = shape[17]
   # neatening 
   remapped_image[27] = shape[0]

   return remapped_image

"""
MAIN CODE STARTS HERE
"""
# load the input image, resize it, and convert it to grayscale
image = cv2.imread("images/faceCM1.jpg")
image = imutils.resize(image, width=500)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

out_face = np.zeros_like(image)

# initialize dlib's face detector (HOG-based) and then create the facial landmark predictor
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(SHAPE_PREDICTOR)

# detect faces in the grayscale image
rects = detector(gray, 1)

# loop over the face detections
for (i, rect) in enumerate(rects):
   """
   Determine the facial landmarks for the face region, then convert the facial landmark (x, y)-coordinates to a NumPy array
   """
   shape = predictor(gray, rect)
   shape = face_utils.shape_to_np(shape)

   #initialize mask array
   remapped_shape = np.zeros_like(shape) 
   feature_mask = np.zeros((image.shape[0], image.shape[1]))   

   # we extract the face
   remapped_shape = face_remap(shape)
   cv2.fillConvexPoly(feature_mask, remapped_shape[0:27], 1)
   feature_mask = feature_mask.astype(np.bool)
   out_face[feature_mask] = image[feature_mask]
   cv2.imshow("mask_inv", out_face)
   cv2.imwrite("out_face.png", out_face)

sample image of cropped face showing the issue

Answer 1

Its because the face shape you are providing is not convex. fillConvexPoly works perfectly on convex shapes only, In this case there is a concave corner (at point #27) and hence the results are messed up.

To fix this, modify the function as

def face_remap(shape):
    remapped_image = cv2.convexHull(shape)
    return remapped_image

This would give you a result which looks like.

Now you may write some more code to remove the triangular section on forehead (if you want it that way)

Answer 2

Using the convex hull formed by the 68 landmarks didn't exactly achieve the desired output, so I had the following approach to this problem using scikit-image instead of OpenCV

1. Load image and predict 68 landmarks

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor('shape_predictor_68_face_landmarks.dat')

img = dlib.load_rgb_image('mean.jpg')

rect = detector(img)[0]
sp = predictor(img, rect)
landmarks = np.array([[p.x, p.y] for p in sp.parts()])

2. Select the landmarks that represents the shape of the face

(I had to reverse the order of the eyebrows landmarks because the 68 landmarks aren't ordered to describe the face outline)

outline = landmarks[[*range(17), *range(26,16,-1)]]

3. Draw a polygon using these landmarks using scikit-image

Y, X = skimage.draw.polygon(outline[:,1], outline[:,0])

4. Create a canvas with zeros and use the polygon as mask to original image

cropped_img = np.zeros(img.shape, dtype=np.uint8)
cropped_img[Y, X] = img[Y, X]

---

For the sake of completeness, I provide below a solution using scipy.spatial.ConvexHull, if this option is still preferred

vertices = ConvexHull(landmarks).vertices
Y, X = skimage.draw.polygon(landmarks[vertices, 1], landmarks[vertices, 0])
cropped_img = np.zeros(img.shape, dtype=np.uint8)
cropped_img[Y, X] = img[Y, X]