Segmentation of images with characters that are joined

Question

I'm trying to draw a boundary box around the different characters in the image below. However, as some of the characters are joined, I can't have the box drawn. I have tried a few things such as; dilating, eroding and trying different blurs but I can't seem to get them apart. Eroding seems to be the best way to get close but If I continue to keep eroding, the characters can't be recognized.

I'm using the OpenCV library in Java to achieve this.

Mat img = Imgcodecs.imread("test.jpg");

        List<MatOfPoint> contours = new ArrayList<>();
        Mat hierarchy = new Mat();
        Imgproc.cvtColor(img, img, Imgproc.COLOR_BGR2GRAY);
        Imgproc.threshold(img, img, 220, 255, 0);
        Imgproc.erode(img, img, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(3, 3)));
        Imgproc.findContours(img, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_NONE);


        MatOfPoint2f approxCurve = new MatOfPoint2f();

        //For each contour found
        for (int i = 0; i < contours.size(); i++) {
            MatOfPoint2f contour2f = new MatOfPoint2f(contours.get(i).toArray());
            double approxDistance = Imgproc.arcLength(contour2f, true) * 0.02;
            Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance, true);

            MatOfPoint points = new MatOfPoint(approxCurve.toArray());

            Rect rect = Imgproc.boundingRect(points);
            Imgproc.rectangle(img, rect.tl(), rect.br(), new Scalar(255, 255, 255), 1, 8, 0);
        }
        System.out.println(img);
        Imgcodecs.imwrite("o.jpg", img);

Before:

After:

if the digits share so many pixels your approach isn't going to work. suggesting another approach would require more knowledge about the variations in font, scale, rotation,... — Piglet, Apr 04 '19 at 15:25
Try to do an erosion with a bigger structuring size, big enough that the numbers are distinguished. I think you are doing 3x3. That is too small, maybe 7x7. Then do dilation with the same size to bring back the original shape (kind of). A combined erosion+dilation is known as _morphological opening_. Something like `Imgproc.erode(img, img, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7)));` `Imgproc.dilate(img, img, Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7)));` — Tetix, Apr 04 '19 at 15:35
@Piglet Yeah they do share quite a few pixels. I'm using a library to generate these images. The library is called: simplecaptcha ([link]http://simplecaptcha.sourceforge.net/ ) I'm not sure how to check for font, scale etc — poofit, Apr 04 '19 at 15:35
@Tetix I've just tried it and setting it at 7 on both removes the issue completely. I've adjusted the values and 5 is the best I can go. https://i.imgur.com/gW0UNbs.png — poofit, Apr 04 '19 at 15:42
This is the result with 7x7: https://i.imgur.com/Vhq7V4f.png — poofit, Apr 04 '19 at 15:50
if connected components was anything good to solve captchas, captchas were completely useless. I guess you have to dig deeper ;) it's like, hey I want to pick this lock but I can't get my fingers in. can you help me? — Piglet, Apr 04 '19 at 15:51
@Piglet Very well put! I'll keep digging and trying things :) — poofit, Apr 04 '19 at 16:25

Bahramdun Adil · Answer 1 · 2019-04-05T07:10:55.270

One possible way (you could try) will work is if you know the count of the characters, e.g: 5, then you can first bound a rectangle around the segmented contours then, divided the width by 5, and then draw the rectangles with black color on the image, and then the image will be successfully segmented.

Follow the steps below:

cv::namedWindow("result", cv::WINDOW_FREERATIO);
cv::Mat img = cv::imread(R"(D:\A49d6.png)");

// to gray
cv::Mat gray;
cv::cvtColor(img, gray, cv::COLOR_BGR2GRAY);
// remove unvisible noise
cv::threshold(gray, gray, 20, 255, cv::THRESH_BINARY);

// this is to make sure to get the correct bounding rect
cv::morphologyEx(gray, gray, cv::MORPH_CLOSE, cv::getStructuringElement(cv::MORPH_CROSS, cv::Size(5, 1)));

std::vector<std::vector<cv::Point> > contours;
cv::findContours(gray, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_SIMPLE);

// because we know that there is only one contour, so get the first posittion (0 index) from the list
cv::Rect rect = cv::boundingRect(contours[0]);

// calculate the width for each character
int width = int(round(rect.width/5.0));
for(int i(0); i<5; i++) {
    cv::Rect r(rect.x+(width*i), rect.y, width, rect.height);
    cv::rectangle(img, r, cv::Scalar(0));
}

cv::imshow("result", img);
cv::waitKey();

And this is the result:

Now the image is clear enough for recognition.

Segmentation of images with characters that are joined

1 Answers1