OpenCV Notehead detection from note Mat/image

Question

I am doing a project where I need to detect musical elements from stave lines, and I am in the point where I know what duration an note element has (quarter, octet, etc) and I what to detect the center of the note-head so that I can found out what note it is (C, D, etc) based on its location on the stave lines.

The problem that I have is that I don't know exactly where to start. I was thinking about some template-matching using full and empty ovals as a template and the element Mat as a source.

Does anyone has any better and optimal solutions?

Examples of element Mats from where I want to find the note-head:

example1 or example2 or example3

Project on GitHub if anyone is interested https://github.com/AmbroziePaval/OMR

Template matching seems like it should work fine, any particular reason to avoid it? You could also use morphological operations but I'd just stick with template matching, it's simpler and more direct. — alkasm, Dec 11 '17 at 19:42
I will give this a go. Hopefully it will not slow down the program by making a template match for every note. I will return shortly with the status. Thanks — Ambrozie Beniamin Paval, Dec 11 '17 at 19:54
Template matching just goes through the image once. Then threshold the resulting image and you'll get all the places the template matched. — alkasm, Dec 11 '17 at 20:08
I have managed to do an implementation for a single element at a time using template matching. I will add the code as a answer. — Ambrozie Beniamin Paval, Dec 11 '17 at 20:24
Don't run template matching multiple times, it's not at all necessary. Template matching gives you an *image* result, and the extrema of the image give you all the locations a match happens. Usually when finding one template you would just take the min or max (depending on template matching method) of the whole image and call it the correct location. But if you just threshold the resulting image, you'll get all the match locations at once. — alkasm, Dec 11 '17 at 20:30
You are right, good point. Using all the matches I will have a list of all center of images. Next step that I have to find the center on the Rect that I am working on. — Ambrozie Beniamin Paval, Dec 11 '17 at 20:34

score 2 · Answer 1 · answered Dec 11 '17 at 20:30

Implementation using template matching for one element (note) at a time.

Example searches all quarters and draw center points in green.

Code:

public Point getAproximateCenterNoteHeadPoint(Mat noteMat) {
    noteMat.convertTo(noteMat, CvType.CV_32FC1);

    Mat fullNoteHeadMat = Imgcodecs.imread(DatasetPaths.FULL_HEAD_TEMPLATE.getPath());
    if (fullNoteHeadMat.channels() == 3) {
        Imgproc.cvtColor(fullNoteHeadMat, fullNoteHeadMat, Imgproc.COLOR_BGR2GRAY);
    }
    fullNoteHeadMat.convertTo(fullNoteHeadMat, CvType.CV_32FC1);

    Mat result = new Mat();
    result.create(noteMat.width(), noteMat.height(), CvType.CV_32FC1);
    double threshold = 0.7;

    Imgproc.matchTemplate(noteMat, fullNoteHeadMat, result, Imgproc.TM_CCOEFF_NORMED);
    Imgproc.threshold(result, result, threshold, 255, Imgproc.THRESH_TOZERO);

    Core.MinMaxLocResult minMaxLocResult = Core.minMaxLoc(result);
    if (minMaxLocResult.maxVal > threshold) {
        Point maxLoc = minMaxLocResult.maxLoc;
        return new Point(maxLoc.x + fullNoteHeadMat.width() / 2, maxLoc.y + fullNoteHeadMat.height() / 2);
    }
    return null;
}

score 2 · Answer 2 · answered Dec 11 '17 at 22:36

Implementation using template matching for all elements at a time, as @Alexander Reynolds suggested in the comments of the question:

public List<Point> findAllNoteHeadCenters(Mat imageMat, List<Rect> elementRectangles) {
    imageMat.convertTo(imageMat, CvType.CV_32FC1);

    Mat fullNoteHeadMat = Imgcodecs.imread(DatasetPaths.FULL_HEAD_TEMPLATE.getPath());
    if (fullNoteHeadMat.channels() == 3) {
        Imgproc.cvtColor(fullNoteHeadMat, fullNoteHeadMat, Imgproc.COLOR_BGR2GRAY);
    }
    fullNoteHeadMat.convertTo(fullNoteHeadMat, CvType.CV_32FC1);

    Mat result = new Mat();
    result.create(imageMat.width(), imageMat.height(), CvType.CV_32FC1);
    double threshold = 0.75;

    Imgproc.matchTemplate(imageMat, fullNoteHeadMat, result, Imgproc.TM_CCOEFF_NORMED);
    Imgproc.threshold(result, result, threshold, 255, Imgproc.THRESH_TOZERO);

    List<Point> centers = new ArrayList<>();
    Set<Rect> foundCenterFor = new HashSet<>();

    while (true) {
        Core.MinMaxLocResult minMaxLocResult = Core.minMaxLoc(result);
        if (minMaxLocResult.maxVal > threshold) {
            Point maxLoc = minMaxLocResult.maxLoc;
            Optional<Rect> containingRect = getPointContainingRect(maxLoc, elementRectangles);

            if (containingRect.isPresent() && !foundCenterFor.contains(containingRect.get())) {
                centers.add(new Point(maxLoc.x + fullNoteHeadMat.width() / 2, maxLoc.y + fullNoteHeadMat.height() / 2));
                foundCenterFor.add(containingRect.get());
            }
            Imgproc.floodFill(result, new Mat(), minMaxLocResult.maxLoc, new Scalar(0));
        } else {
            break;
        }
    }
    return centers;
}

score 1 · Accepted Answer · answered Dec 13 '17 at 10:35

Try using Chamfer based distance transform transform to find the center of your point. The algorithm passes the image 2 times to calculate the distance of each object pixel to the nearest margin. The center point of your object will be the one with the greatest distance assigned.

OpenCV Notehead detection from note Mat/image

3 Answers3