Text Segmentation on Texts Occluded by Objects

Question

I'm working on a project to extract characters of a license plate. I have developed a method by MSER to segment characters for license plates with complex background and it works well. The problem is, in some cases like seen below, the frame(license plate holder) of license plate concatenates or occupies a part of characters. By having almost the same color, the characters and the frame come as a single object. It is impossible to extract these characters from frame so I fail to detect characters.

I looked around if "Horizontal Projection" of license plate could yield something useful but it seems it also requires a good skew correction algorithm before applying it, which may not be the best solution to handle this problem by needing a series of new algorithms. Thus I wanted to ask here if such a good way exist and you point me the proper way.

Thanks in advance.

(ps: I blurred a part of license plates in order to protect privacy. Quality of images are not satifying but I think it is enough to understand the problem)

Answer 1

You could try limiting the region of interest with a logical mask before segmenting the characters. Let's make one with a convex hull, they are really versatile:

Using Matlab 2016b:

Plate = imread(Plate.jpg);

grayPlate = rgb2gray(Plate); % rgb -> grayscale

bwPlate = imbinarize(grayPlate); % binarize, Otsu's method.

bwPlate = imopen(bwPlate, strel('disk', 4)); 
% Morphological opening, removes small white areas. These bloat the convex
% hull if let through.

convPlate = bwconvhull(bwPlate);

for i=1:3 %Apply the logical mask
    tempPlate = Plate(:,:,i);
    tempPlate(~convPlate) = 255;
    Plate(:,:,i) = tempPlate;
end

Result :

The car pros made our life a bit harder. But this should be a lot easier to manage, especially if you fiddle around with the grayscale weights, binarization and filtering.