Approach #1
One strategy would be to label the image, get the label for one of the corners, as one sample for the background color and create a mask of that color. So, the inverted mask area is the one we are trying to crop into. For the cropping, we will take inspiration from Crop black border of image using NumPy : Answer post
.
Here's the implementation -
import cv2
from scipy.ndimage import label
im = cv2.imread('xYTKn.png') # read in input image
L,_ = label(im)
mask = (L!=L[0,0]).any(-1)
out = im[np.ix_(mask.any(1), mask.any(0))]
cv2.imwrite('output.png',out)
Input -

Output -

Approach #2
Another similar strategy would be to convert to grayscale upfront, then label and get the required mask, like so -
gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
L,_ = label(gray)
mask = (L!=L[0,0])
out = im[np.ix_(mask.any(1), mask.any(0))]
Benchmarking
We will time the proposed solutions on the given sample image. The mask creation is different between the two approaches. So, we will time those sections only -
In [70]: %%timeit
...: L,_ = label(im)
...: mask = (L!=L[0,0]).any(-1)
47.4 ms ± 2.37 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [71]: %%timeit
...: gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
...: L,_ = label(gray)
...: mask = (L!=L[0,0])
2.59 ms ± 20.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)