You could use a flood fill starting at the bottom-right corner to fill all pixels less than some "fuzziness" distance (in shades of grey terms rather than in geometric distance terms) with black so they all come out to the same class.
Here I do it with ImageMagick just in Terminal, and colour using red and blue, rather than black, to show the concept:
convert input.jpg -fuzz 15% -fill red -floodfill +1140+760 black result15.jpg

Or, allowing slightly fewer colours (darker) to match via fuzziness:
convert input.jpg -fuzz 10% -fill blue -floodfill +1140+760 black result10.jpg

You can do this with OpenCV in Python, and Wand and other tools. Here is an example showing how to do a floodfill withPIL/Pillow.