Optimizing performance of GrabCut in opencv-java

Question

Recently I have been given a project, where I have to extract face (face+hair) from a given image.

I am solving this problem in the following ways.

1. I am extracting face locations from given image. [I am getting a rectangle]
2. I am extracting that rectangle and placing it in another image of same dimensions as input image.[face_image]
3. I am applying grabCut algorithm on the face_image of step 2.

When the face_image contains smooth background then the algorithm grabCut it working well but when the background of face_image is complex then the algorithm grabCut extracts some part of background too in the processed image.

Here is a snapshot of the results that I am getting.

Here is my code of grabCut:

public void extractFace(Mat image, String fileNameWithCompletePath, 
                       int xOne, int xTwo, int yOne, int yTwo) throws CvException {

    System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

    Rect rectangle = new Rect(xOne, yOne, xTwo, yTwo);
    Mat result = new Mat();
    Mat bgdModel = new Mat();
    Mat fgdModel = new Mat();
    Mat source = new Mat(1, 1, CvType.CV_8U, new Scalar(3));
    Imgproc.grabCut(image, result, rectangle, bgdModel, fgdModel, 8, Imgproc.GC_INIT_WITH_RECT);
    Core.compare(result, source, result, Core.CMP_EQ);
    Mat foreground = new Mat(image.size(), CvType.CV_8UC3, new Scalar(255, 255, 255));
    image.copyTo(foreground, result);
    Imgcodecs.imwrite(fileNameWithCompletePath, foreground);
}

How can I improve performance of grabCut algorithm so that it will detect only face and hair from given image?

Answer 1

You should be able to do this by "helping" grabCut know a little about the foreground and background. There is a python tutorial that shows how this is done manually by selecting the foreground and background.

To do this automatically, you will need to find programmatic ways to detect the foreground and background. The foreground consists mostly of hair and skin so you will need to detect them.

Skin - There are several papers and blogs on how to do this. Some of them are pretty simple and this OpenCV tutorial may also help. I've found plain hue/saturation to get me pretty far.

Hair - This is trickier but is definitely still doable. You may be able to hair and just use skin and background if this turns out to be too much work.

Background - You should be able to use range() to find things in the image that are purple, green, and blue. You know for sure that these things are not skin or hair and are therefore part of the background.

Use thresholding to create a mask of the areas that are most likely skin, hair, and background. You can then use them as bgdModel and fgdModel (or the skin and hair masks) instead of Mat().

Sorry this is so high-level. I hope it helps!

Answer 2

Another approach, since you have already detected the face, is to simply choose a better initial mask for initialising GrabCut - e.g. by using an oval instead of a rectangle.

1. Detect face rectangle (as you are already doing)

2. Create a mask:

   a) Create a new black image of the same size as your input image

   b) Draw a white-filled ellipse with the same height, width, top and left positions as the face rectangle

3. Call GrabCut with GC_INIT_WITH_MASK instead of GC_INIT_WITH_RECT:

   Imgproc.grabCut(image, mask, rectangle, bgdModel, fgdModel, 8, Imgproc.GC_INIT_WITH_MASK);
   

This initializes the foreground with a better model because faces are more oval-shaped than rectangle-shaped, so it should include less of the background to begin with.

Answer 3

I would suggest to "play" with the rectangle coordinates (int xOne, int xTwo, int yOne, int yTwo). Using your code and these coordinates 1, 400, 30, 400 I was able to avoid the background. (I tried to post the images I successfully cropped but I need at least 10 reputation to do so)

Answer 4

The best optimization that can be done to any Java routine is conversion to a native language.