OpenCV: Why does Haar classifier only detect one face and one eye?

Question

I am an absolutely newcomer to OpenCV and Haar classifiers. I have copied the following code and it basically works fine, but it only returns one face and one eye. What about the second eye and another face?

    const char[] eyesCascadeFilename = "haarcascade_eye.xml";
    const char[] faceCascadeFilename = "haarcascade_frontalface_alt2.xml";
    const int HaarOptions = CV_HAAR_DO_ROUGH_SEARCH;

    // prepare the image for fast processing
    Mat grayImage;
    cvtColor(colorImage, grayImage, CV_BGR2GRAY);
    equalizeHist(grayImage, grayImage);

    // detect the faces on the image
    std::vector<cv::Rect> faces;
    faceCascade.detectMultiScale(grayImage, faces, 1.1, 2, HaarOptions, cv::Size(60, 60));

    for (int i = 0; i < faces.size(); i++) {
        // visualize the faces
        cv::Point pt1(faces[i].x + faces[i].width, faces[i].y + faces[i].height);
        cv::Point pt2(faces[i].x, faces[i].y);
        cv::rectangle(colorImage, pt1, pt2, cvScalar(0, 255, 0, 0), 1, 8 ,0);
        // detect the eyes within the facial roi
        cv::Rect rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
        cv::Mat roi = grayImage(rect);
        std::vector<cv::Rect> eyes;
        // here's a problem ...:
        eyesCascade.detectMultiScale(roi, eyes, 1.1, 2, HaarOptions, cv::Size(30, 30));
        //eyesCascade.detectMultiScale(roi, eyes);
        for (int i = 0; i < eyes.size(); i++) {
            // visualize the eyes
            cv::Point pt1(faces[i].x + eyes[i].x + eyes[i].width, faces[i].y + eyes[i].y + eyes[i].height);
            cv::Point pt2(faces[i].x + eyes[i].x, faces[i].y + eyes[i].y);
            cv::rectangle(colorImage, pt1, pt2, cvScalar(0, 210, 255, 0), 1, 8 ,0);
        }

    }

Here's what it looks like:

Answer 1

EDIT

In your code, in the inner for loop, you are actually making a new variable with the same name as the iteration variable of the outer for loop (i). Since they are in different scopes this is allowed, and the variable in the inner scope "owns" the name. You will not be able to access the outer-scoped i inside the inner scope.

The for loop declaration itself is part of the scope of the for loop, so counts as part of the inner-scope in the case of the second i.

In your case, this translates into the fact that you will only be able to draw one eye at a time for a given face, even if two were detected. You can't reach eye(2) for face(1). You are actually lucky your code didn't crash: this is only due to the fact that both your face and eye vector are of the same size (2).

In debug mode you got eye.size == 1 probably just because at a particular moment the algorithm was only able to detect one eye. It is likely that at the next iteration eye.size would have been 2.

So, use, let's say, int j = 0 in your inner loop to iterate through the eye vector, and you'll be fine!

END EDIT

This is my own (working) version of this algorithm:

#include "FaceDetectionHaar.h"


FaceDetectionHaar::FaceDetectionHaar()
{
    face_cascade_name = "haarcascade_frontalface_alt.xml";
    eyes_cascade_name = "haarcascade_eye_tree_eyeglasses.xml";

    if( !face_cascade.load( face_cascade_name ) ){ printf("--(!)Error loading\n");};
    if( !eyes_cascade.load( eyes_cascade_name ) ){ printf("--(!)Error loading\n");};
}

void FaceDetectionHaar::execute(IplImage in)
{
    Mat imgMat(&in);

    std::vector<Rect> faces;
    Mat frame_gray;

    cvtColor( imgMat, frame_gray, CV_BGR2GRAY );
    equalizeHist( frame_gray, frame_gray );

    //-- Detect faces
    face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );

    for( int i = 0; i < faces.size(); i++ )
    {
        Point center( faces[i].x + faces[i].width*0.5, faces[i].y + faces[i].height*0.5 );
        ellipse( imgMat, center, Size( faces[i].width*0.5, faces[i].height*0.5), 0, 0, 360, Scalar( 255, 0, 255 ), 4, 8, 0 );

        Mat faceROI = frame_gray( faces[i] );
        std::vector<Rect> eyes;

        //-- In each face, detect eyes
        eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CV_HAAR_SCALE_IMAGE, Size(30, 30) );

        for( int j = 0; j < eyes.size(); j++ )
        {
            Point center( faces[i].x + eyes[j].x + eyes[j].width*0.5, faces[i].y + eyes[j].y + eyes[j].height*0.5 );
            int radius = cvRound( (eyes[j].width + eyes[i].height)*0.25 );
            circle( imgMat, center, radius, Scalar( 255, 0, 0 ), 4, 8, 0 );
        }
    }
    in = imgMat;
}

The code is way simpler than yours for the ROI part. And maybe the use of the Haar option CV_HAAR_SCALE_IMAGE instead of CV_HAAR_DO_ROUGH_SEARCH will help.

This code is for the most part a copy of the one given on the OpenCV website. You should try the Haar Cascades xml files they provide too, both for faces and eyes.

Answer 2

Okay I now solved the problem with my algorithm too... Very akward, I used for the eyes loop the same iteration variable "i" like for the outer loop to iterate through the faces... :-/ Strange that the compiler didn't gave me an error that variable "i" is already defined...!

Here's the working code:

    const char[] eyesCascadeFilename = "haarcascade_eye.xml";
    const char[] faceCascadeFilename = "haarcascade_frontalface_alt2.xml";
    const int HaarOptions = CV_HAAR_DO_ROUGH_SEARCH;

    // prepare the image for fast processing
    Mat grayImage;
    cvtColor(colorImage, grayImage, CV_BGR2GRAY);
    equalizeHist(grayImage, grayImage);

    // detect the faces on the image
    std::vector<cv::Rect> faces;
    faceCascade.detectMultiScale(grayImage, faces, 1.1, 2, HaarOptions, cv::Size(60, 60));

    for (int i = 0; i < faces.size(); i++) {
        // visualize the faces
        cv::Point pt1(faces[i].x + faces[i].width, faces[i].y + faces[i].height);
        cv::Point pt2(faces[i].x, faces[i].y);
        cv::rectangle(colorImage, pt1, pt2, cvScalar(0, 255, 0, 0), 1, 8 ,0);
        // detect the eyes within the facial roi
        cv::Rect rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
        cv::Mat roi = grayImage(rect);
        std::vector<cv::Rect> eyes;
        // here's a problem ...:
        eyesCascade.detectMultiScale(roi, eyes, 1.1, 2, HaarOptions, cv::Size(30, 30));
        //eyesCascade.detectMultiScale(roi, eyes);
        for (int j = 0; j < eyes.size(); j++) {
            // visualize the eyes
            cv::Point pt1(faces[i].x + eyes[j].x + eyes[j].width, faces[i].y + eyes[j].y + eyes[j].height);
            cv::Point pt2(faces[i].x + eyes[j].x, faces[i].y + eyes[j].y);
            cv::rectangle(colorImage, pt1, pt2, cvScalar(0, 210, 255, 0), 1, 8 ,0);
        }

    }