How to detect the outline of a shape and then crop the shape from image?

Question

• I am attempting to only keep the part of the image bounded by the orange/greenish line in lot #17.

• As you can see the shape is fairly non standard and I am new to image processing so my approach thus far has been brute forced and error prone.

• Each image I need to do this for has a black dot (rgb of (77,77,77)) in the center of the shape I want to crop which has been my anchor.

import PIL
import pandas as pd

image = PIL.Image.open(file)
rgb_im = image.convert('RGB')

color = (77,77,77)
colorindex = pd.DataFrame(data = None,columns = ['X','Y'])
for x in range(image.size[0]):
 for y in range(image.size[1]):
    r, g, b = rgb_im.getpixel((x, y))
    if (r,g,b) == color:
        append = [x,y]
        append = pd.Series(append,index = colorindex.columns)
        colorindex = colorindex.append(append,ignore_index = True)
center = [colorindex.mode()['X'][0],colorindex.mode()['Y'][0]] 

line = pd.read_excel('C:/Users/lines RGb.xlsx') ##Prerecorded RGB Values

def findparcelline(CenterX,CenterY,direction):

if direction == 'left':
    for x in range(CenterX):
        r,g,b = rgb_im.getpixel((CenterX-x,CenterY))
        for i in range(len(line)):
            if (r,g,b) == (line.loc[i][0],line.loc[i][1],line.loc[i][2]):
                pixelsave = CenterX-x
                return pixelsave

elif direction == 'right':
    for x in range(CenterX):
        r,g,b = rgb_im.getpixel((CenterX+x,CenterY))
        for i in range(len(line)):
            if (r,g,b) == (line.loc[i][0],line.loc[i][1],line.loc[i][2]):
                pixelsave = CenterX+x
                return pixelsave

elif direction == 'down':
    for y in range(CenterY):
        r,g,b = rgb_im.getpixel((CenterX,CenterY + y))
        for i in range(len(line)):
            if (r,g,b) == (line.loc[i][0],line.loc[i][1],line.loc[i][2]):
                pixelsave = CenterY + y
                return pixelsave

elif direction == 'up':
    for y in range(CenterY):
        r,g,b = rgb_im.getpixel((CenterX,CenterY - y))
        for i in range(len(line)):
            if (r,g,b) == (line.loc[i][0],line.loc[i][1],line.loc[i][2]):
                pixelsave = CenterY - y
                return pixelsave

directions = ['left','down','right','up']
coords =[]
for direction in directions:
 coords.append(findparcelline(center[0],center[1],direction))       
im1 = image.crop(coords)

• My code only works for right side up rectangular shapes (which a good bit of them are) but it will fail when it comes to something like in the example.
• I've thought about using the code written this far to then 'walk the line' from the pixel location provided via a 9x9 array of pixels and only selecting the ones that:

1. aren't previously selected
2. match the prerecorded color values
3. are closest to the anchor pixel location

• But in the example there are even more rgb color values to and even some holes in the line I'm interested in.

• Is there a way to obtain the coordinates of the line bounding the black dot in the center and subsequently crop the image after having recording all the coordinates?

Thanks in advance.

Answer 1

First of all: If you have access to the generation of these images, save them as lossless PNGs! Those JPG artifacts make it even harder to get proper results. For example, only one pixel of your "black" dot actually has RGB values of (77, 77, 77). Therefore, I omitted the programmatically finding of the "black" dot, and assumed the image center as the dot location.

Since you have kind of red-ish lines with some kind of yellow-ish dots, I rectified the red channel by subtracting a portion of the green channel to get rid of yellow-ish colors. After some further emphasizing (red-ish lines have high values in the red channel), the new red channel looks like this:

On that new red channel, I use some kind of Laplace operator to detect the (red-ish) lines. After some further processing, that'd be the result:

From there, it's just some thresholding using Otsu's method to get a proper binary image to work on:

Finally, I find all contours, and iterate them. If I find an inner(!) contour – please see this answer for an extensive explanation on contour hierarchies – which contains the location of the "black" dot, that must be shape of interest. Since you might get some odd, open contours from the surrounding, you need to stick to inner contours. Also, it's an assumption here, that the shape of interest is closed.

After extracting the proper contour, you just need to set up a proper mask, and for example blacken the background, or crop the image using the bounding rectangle of that mask:

Here's the full code:

import cv2
import numpy as np

# Read image, split color channels
img = cv2.imread('5aY7A.jpg')
b, g, r = cv2.split(img)

# Rectify red-ish lines (get rid of yellow-ish dots) by subtracting
# green channel from red channel
r = r - 0.5 * g
r[r < 0] = 0

# Emphasize red-ish lines
r **= 2
r = cv2.normalize(r, 0, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)

# Detection of red-ish lines by Laplace operator
r = cv2.Laplacian(r, cv2.CV_64F)
r = cv2.erode(r, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))
r = cv2.GaussianBlur(r, (5, 5), 0)
r = cv2.normalize(r, 0, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)

# Mask red-ish lines
r = cv2.threshold(r, 10, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]
r = cv2.morphologyEx(r, cv2.MORPH_CLOSE,
                     cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))

# Detection of "black" dot location omitted here due to JPG artifacts...
dot = (916, 389)

# Find contours from masked red-ish lines
cnts, hier = cv2.findContours(r, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

# Find some inner(!) contour containing the "black dot"
cnt = None
for i, c in enumerate(cnts):
    if cv2.pointPolygonTest(c, dot, True) > 0 and hier[0, i, 3] != -1:
        cnt = c
        break

if cnt is None:
    print('Something went wrong, no contour found.')
else:
    mask = cv2.drawContours(np.zeros_like(r), [cnt], -1, 255, cv2.FILLED)
    output = cv2.bitwise_xor(img, np.zeros_like(img), mask=mask)
    cv2.imshow('Output', output)
    cv2.waitKey(0)

cv2.destroyAllWindows()

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.19041-SP0
Python:        3.9.1
PyCharm:       2021.1.2
NumPy:         1.20.3
OpenCV:        4.5.2
----------------------------------------