Braille Recognition from an image using OpenCV/C++

Question

I am aiming to develop an iOS app that captures an image > extracts Braille dots represented in a specific color (Blue) > translates Braille letters into text using image processing techniques.

My approach was to use OpenCV/C++ to have the image be processed to extract the blue colored dots as shown in this photo into this photo

The next step is to recognize Braille in the image to translate it into text, one solution was to put a grid on the image to find the intersection points pixel color value then classify them into (1 if white , 0 if black) as represented in this photo

The problem with the proposed solution was:

1. how to position the grid rows/columns at the wanted positions?
2. how to get the coordinates and value(0 or 1)of the intersection points?

If you have any suggestion/solution about the proposed solution or any other solution please share them It would be appreciated since I don't have an experience in OpenCV/C++ field.

*note that python solutions cannot be used in iOS (as far as I know).

I attached my code for reference

 + (UIImage *)detectRedShapesInImage:(UIImage *)image{
cv::Mat mat;
UIImageToMat(image, mat);
cv::medianBlur(mat, mat, 3);

// Convert input image to HSV
cv::Mat hsv_image;
cv::cvtColor(mat, hsv_image, cv::COLOR_BGR2HSV);

// Threshold the HSV image, keep only the red (replaced it with blue) pixels
cv::Mat lower_red_hue_range;
cv::Mat upper_red_hue_range;
cv::inRange(hsv_image, cv::Scalar(0, 100, 100), cv::Scalar(10, 255, 255), lower_red_hue_range);
cv::inRange(hsv_image, cv::Scalar(160, 100, 100), cv::Scalar(179, 255, 255), upper_red_hue_range);

// Combine the above two images
cv::Mat red_hue_image;
cv::addWeighted(lower_red_hue_range, 1.0, upper_red_hue_range, 1.0, 0.0, red_hue_image);
cv::GaussianBlur(red_hue_image, red_hue_image, cv::Size(9, 9), 2, 2);


// detect circules, for now it doesnot take all circles
std::vector<cv::Vec4f> circles;
cv::HoughCircles(red_hue_image, circles, cv::HOUGH_GRADIENT, 1.0, 20,  150, 40, 0, 0);

// Loop over all detected circles and outline them on the original image
if(circles.size() == 0) std::exit(-1);
for(size_t current_circle = 0; current_circle < circles.size(); ++current_circle) {
    cv::Point center(std::round(circles[current_circle][0]), std::round(circles[current_circle][1]));
    int radius = std::round(circles[current_circle][2]);
    
    cv::circle(red_hue_image, center, radius, cv::Scalar(0, 255, 0), 5);
    
}


UIImage *maskedShapesImg = MatToUIImage(red_hue_image);

return maskedShapesImg;}

Answer 1

Hints:

Assuming that the text is fairly horizontal and the spacing between lines is sufficient:

• get the centers of the dots;
• find the shortest horizontal and vertical distances between dots; these give you the horizontal and vertical pitch;
• cluster the dots that are no more than one horizontal or two vertical pitches apart (with a safety margin); a cluster should correspond to a single character;
• find the top-left corner of the characters;
• find the median horizontal and vertical distance between the characters;
• from this information (dot and character spacing), map the dots to the grid by predicting the grid nodes and using the nearest-neighbor rule.

This work is made a little tricky by the fact that not all character occupy the two columns and three rows of dots.