How can I cut a green background with the foreground from the rest of the picture in Python?

Question

I'm trying to cut multiple images with a green background. The center of the pictures is green and i want to cut the rest out of the picture. The problem is, that I got the pictures from a video, so sometimes the the green center is bigger and sometimes smaller. My true task is to use K-Means on the knots, therefore i have for example a green background and two ropes, one blue and one red.

I use python with opencv, numpy and matplotlib.

I already cut the center, but sometimes i cut too much and sometimes i cut too less. My Imagesize is 1920 x 1080 in this example.

Here the knot is left and there is more to cut

Here the knot is in the center

Here is another example

Here is my desired output from picture 1

Example 1 which doesn't work with all algorithm

Example 2 which doesn't work with all algorithm

Example 3 which doesn't work with all algorithm

Here is my Code so far:

import numpy as np
import cv2
import matplotlib.pyplot as plt
from PIL import Image, ImageEnhance

img = cv2.imread('path')

print(img.shape)

imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

crop_img = imgRGB[500:500+700, 300:300+500]

plt.imshow(crop_img)
plt.show()

Answer 1

You can change color to hsv.

src = cv2.imread('path')
imgRGB = cv2.cvtColor(src, cv2.COLOR_BGR2RGB)
imgHSV = cv2.cvtColor(imgRGB, cv2.COLOR_BGR2HSV)

Then use inRange to find only green values.

lower = np.array([20, 0, 0])    #Lower values of HSV range; Green have Hue value equal 120, but in opencv Hue range is smaler [0-180]
upper = np.array([100, 255, 255])  #Uppervalues of HSV range
imgRange = cv2.inRange(imgHSV, lower, upper)

Then use morphology operations to fill holes after not green lines

#kernels for morphology operations
kernel_noise = np.ones((3,3),np.uint8) #to delete small noises
kernel_dilate = np.ones((30,30),np.uint8)  #bigger kernel to fill holes after ropes
kernel_erode = np.ones((38,38),np.uint8)  #bigger kernel to delete pixels on edge that was add after dilate function

imgErode = cv2.erode(imgRange, kernel_noise, 1)
imgDilate = cv2.dilate(imgErode , kernel_dilate, 1)
imgErode = cv2.erode(imgDilate, kernel_erode, 1)

Put mask on result image. You can now easly find corners of green screen (findContours function) or use in next steps result image

res = cv2.bitwise_and(imgRGB, imgRGB, mask = imgErode)  #put mask with green screen on src image

Answer 2

First step is to extract green channel from your image, this is easy with OpenCV numpy and would produce grayscale image (2D numpy array)

import numpy as np
import cv2
img = cv2.imread('knots.png')
imgg = img[:,:,1] #extracting green channel 

Second step is using thresholding, which mean turning grayscale image into binary (black and white ONLY) image for which OpenCV has ready function: https://docs.opencv.org/3.4.0/d7/d4d/tutorial_py_thresholding.html

imgt = cv2.threshold(imgg,127,255,cv2.THRESH_BINARY)[1]

Now imgt is 2D numpy array consisting solely of 0s and 255s. Now you have to decide how you would look for places of cuts, I suggest following:

• topmost row of pixel containing at least 50% of 255s
• bottommost row of pixel containing at least 50% of 255s
• leftmost column of pixel containing at least 50% of 255s
• rightmost column of pixel containing at least 50% of 255s

Now we have to count number of occurences in each row and each column

height = img.shape[0]
width = img.shape[1]
columns = np.apply_along_axis(np.count_nonzero,0,imgt)
rows = np.apply_along_axis(np.count_nonzero,1,imgt)

Now columns and rows are 1D numpy arrays containing number of 255s for each column/row, knowing height and width we could get 1D numpy arrays of bool values following way:

columns = columns>=(height*0.5)
rows = rows>=(width*0.5)

Here 0.5 means 50% mentioned earlier, feel free to adjust that value to your needs. Now it is time to find index of first True and last True in columns and rows.

icolumns = np.argwhere(columns)
irows = np.argwhere(rows)
leftcut = int(min(icolumns))
rightcut = int(max(icolumns))
topcut = int(min(irows))
bottomcut = int(max(irows))

Using argwhere I got numpy 1D arrays of indexes of Trues, then found lowest and greatest. Finally you can clip your image and save it

imgout = img[topcut:bottomcut,leftcut:rightcut]
cv2.imwrite('out.png',imgout)

There are two places which might be requiring adjusting: % of 255s (in my example 50%) and threshold value (127 in cv2.threshold).

EDIT: Fixed line with cv2.threshold

Answer 3

The code below does what you want. First it converts the image to the HSV colorspace, which makes selecting colors easier. Next a mask is made where only the green parts are selected. Some noise is removed and the rows and columns are summed up. Finally a new image is created based on the first/last rows/cols that fall in the green selection.

Since in all provided examples a little extra of the top needed to be cropped off I've added code to do that. First I've inverted the mask. Now you can use the sum of the rows/cols to find the row/col that is fully within the green selection. It is done for the top. In the image below the window 'Roi2' is the final image.

Edit: updated code after comment by ts.
Updated result:

Code:

import numpy as np 
import cv2

# load image
img = cv2.imread("gr.png")
# convert to HSV
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) 
# set lower and upper color limits
lower_val = (30, 0, 0)
upper_val = (65,255,255)
# Threshold the HSV image to get only green colors
# the mask has white where the original image has green
mask = cv2.inRange(hsv, lower_val, upper_val)
# remove noise
kernel =  np.ones((8,8),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

# sum each row and each volumn of the image
sumOfCols = np.sum(mask, axis=0)
sumOfRows = np.sum(mask, axis=1)

# Find the first and last row / column that has a sum value greater than zero, 
# which means its not all black. Store the found values in variables
for i in range(len(sumOfCols)):
    if sumOfCols[i] > 0:
        x1 = i
        print('First col: ' + str(i))
        break

for i in range(len(sumOfCols)-1,-1,-1):
    if sumOfCols[i] > 0:
        x2 = i
        print('Last col: ' + str(i))
        break

for i in range(len(sumOfRows)):
    if sumOfRows[i] > 0:
        y1 = i
        print('First row: ' + str(i))
        break

for i in range(len(sumOfRows)-1,-1,-1):
    if sumOfRows[i] > 0:
        y2 = i
        print('Last row: ' + str(i))
        break

# create a new image based on the found values
#roi = img[y1:y2,x1:x2]

#show images
#cv2.imshow("Roi", roi)



# optional: to cut off the extra part at the top:
#invert mask, all area's not green become white
mask_inv = cv2.bitwise_not(mask)
# search the first and last column top down for a green pixel and cut off at lowest common point
for i in range(mask_inv.shape[0]):
    if mask_inv[i,0] == 0 and mask_inv[i,x2] == 0:
        y1 = i
        print('First row: ' + str(i))
        break

# create a new image based on the found values
roi2 = img[y1:y2,x1:x2]

cv2.imshow("Roi2", roi2)
cv2.imwrite("img_cropped.jpg", roi2)
cv2.waitKey(0)
cv2.destroyAllWindows()

Answer 4

Based on the new images you added I assume that you do not only want to cut out the non green parts as you asked, but that you want a smaller frame around the ropes/knot. Is that correct? If not, you should upload the video and describe the purpose/goal of the cropping a bit more, so that we can better help you.

Assuming you want a cropped image with only the ropes, the solution is quite similar the the previous answer. However, this time the red and blue of the ropes are selected using HSV. The image is cropped based on the resulting mask. If you want the image somewhat bigger than just the ropes, you can add extra margins - but be sure to account/check for the edge of the image.

Note: the code below works for the images that that have a full green background, so I suggest you combine it with one of the solutions that only selects the green area. I tested this for all your images as follows: I took the code from my other answer, put it in a function and added return roi2 at the end. This output is fed into a second function that holds the code below. All images were processed successful.

Result:

Code:

import numpy as np 
import cv2

# load image
img = cv2.imread("image.JPG")
# blue
lower_val_blue = (110, 0, 0)
upper_val_blue = (179,255,155)
# red
lower_val_red = (0, 0, 150)
upper_val_red = (10,255,255)
# Threshold the HSV image
mask_blue = cv2.inRange(img, lower_val_blue, upper_val_blue)
mask_red = cv2.inRange(img, lower_val_red, upper_val_red)
# combine masks
mask_total = cv2.bitwise_or(mask_blue,mask_red)

# remove noise
kernel =  np.ones((8,8),np.uint8)
mask_total = cv2.morphologyEx(mask_total, cv2.MORPH_CLOSE, kernel)

# sum each row and each volumn of the mask
sumOfCols = np.sum(mask_total, axis=0)
sumOfRows = np.sum(mask_total, axis=1)

# Find the first and last row / column that has a sum value greater than zero, 
# which means its not all black. Store the found values in variables
for i in range(len(sumOfCols)):
    if sumOfCols[i] > 0:
        x1 = i
        print('First col: ' + str(i))
        break

for i in range(len(sumOfCols)-1,-1,-1):
    if sumOfCols[i] > 0:
        x2 = i
        print('Last col: ' + str(i))
        break

for i in range(len(sumOfRows)):
    if sumOfRows[i] > 0:
        y1 = i
        print('First row: ' + str(i))
        break

for i in range(len(sumOfRows)-1,-1,-1):
    if sumOfRows[i] > 0:
        y2 = i
        print('Last row: ' + str(i))
        break

# create a new image based on the found values
roi = img[y1:y2,x1:x2]

#show image
cv2.imshow("Result", roi)
cv2.imshow("Image", img)

cv2.waitKey(0)
cv2.destroyAllWindows()