detecting blurred rectangles on multimodal grayscaleimage

Question

I have the following image

I'm trying to find the pixel coordinates of main rectangles (those between white lines). I tried few things but I can't obtain good enough solution. The solution doesn't have to be perfect and it's ok if not all rectangles are detected (especially those really small ones). Though corners location will have to be as much exact as possible, especially with those bigger blurry (I'm trying to write some simple AR engine).

I can clarify there are only 4 levels of grayscale: 0, 110, 180 and 255 (when printing, no screen it will vary because of lightning and shadows)

So far I tried few things:

1. manual multilevel thresholding (because of shadows and different lightning it didn't work)
2. adaptive thresholding : 2 problems:
   • it combines 180 and 255 colors into white, and 0, 110 into black
   • edge/corner location of blurred(bigger) rectangles is not exact (it adds blur to rectangle area)
3. sobel edge detection (corners of blurred rectangles are more sharp, but it detects also inner edges in rectangles, also those edge contours are not always closed

Looks like combining those two thinks somehow would yield better results. Or maybe somebody have different idea?

I was also thinking about doing floodfill, but it was hard to find for sure good seed point and threshold automatically (there might be some other white objects in the background). Besides I will want to optimize later this for GPU and floodfill algorithm is rather not a good fit for this.

Below is some sample code I tried so far:

image = cv2.imread('data/image.jpg');
gray  = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
cv2.imshow('image', gray) 


adaptive = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 601, 0)
cv2.imshow('adaptive', adaptive)


gradx = cv2.Sobel(gray, ddepth=cv2.CV_32F, dx=1, dy=0, ksize=3)
grady = cv2.Sobel(gray, ddepth=cv2.CV_32F, dx=0, dy=1, ksize=3)
abs_gradx = cv2.convertScaleAbs(grady)
abs_grady = cv2.convertScaleAbs(grady)
grad = cv2.addWeighted(abs_gradx, 0.5, abs_grady, 0.5, 0)

kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(5,5))
grad = cv2.morphologyEx(grad, cv2.MORPH_OPEN, kernel)
grad = cv2.morphologyEx(grad, cv2.MORPH_CLOSE, kernel)
cv2.imshow('sobel',grad)


#kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(7,7))
#grad = cv2.morphologyEx(grad, cv2.MORPH_OPEN, kernel)
retval, grad = cv2.threshold(grad, 10, 255, cv2.THRESH_BINARY)
cv2.imshow('sobel+morph+thrs',grad)

cv2.waitKey()

Answer 1

I believe your answer lies in using the Hough transform to detect lines, extending these lines to span the breaks between the darker squares, and then looking for intersects marking out corners. I've had a quick play in Matlab and have come up with the following, it's not perfect but should show the potential:

% Open image
i = imread('https://i.stack.imgur.com/kwcXm.jpg');

% Use a sharpening filter to enhance some of the edges
H = fspecial('unsharp');
i = imfilter(i, H, 'replicate');
% Detect edge segments using canny
BW = edge(i, 'canny');

% Apply hough transform to edges
[H, T, R] = hough(BW, 'RhoResolution', 0.5, 'Theta', -90:0.5:89.5);
% Find peaks in hough transform
P = houghpeaks(H, 5, 'threshold', ceil(0.1*max(H(:))));
% Extract lines from peaks, extending partial lines
lines = houghlines(BW, T, R, P, 'FillGap', 100, 'MinLength', 5);

% Plot detected lines on image
imshow(i); hold on;
for k = 1:length(lines)
   xy = [lines(k).point1; lines(k).point2];
   plot(xy(:,1),xy(:,2),'LineWidth',2,'Color','green');
end

With the final result:

Obviously there's room for improvement, with a number of lines still to detect, but if tweaking the various parameters doesn't work you could take the initial result and the search for more lines with similar angles to obtain a more complete set. The corners can then be found from the intersects which should be simple enough to extract.

Answer 2

I would try the following:

• Hough transform to detect all straight lines
• Look for sets of parallel lines that are:
  • A sufficient distance apart
  • Separated by the lighter color

There are a couple of things that make your problem trickier than it needs to be:

• Perspective distortion
• Changes in lighting, minor shadows

If you could minimize the above, it may help solving the problem.