Challenging image segmentation: background and objects are similar

Question

I'm new to image segmentation, but I need to do it to get a database for the machine learning classifier.

Essentially I have a video similar to this image:

My job is to identify cows in the foreground, or at least any cow at all. I realize there is an occlusion problem, but for a starter I'd like to correctly segment a lonely cow, like the one with the red rectangle around it (hand-drawn).

In less challenging problems, such as this, I discriminate by adding a threshold for every pixel, that either becomes (0,0,0) for the object or (255,255,255) for the background:

Then I label the pixels with the same values to get classes and obtain the rectangle for large enough 'blobs'.

For the image above this approach will not work as the objects and the background are similar + there are a lot of shadows, side lighting etc, so I'm not sure how to approach it. Any suggestions are welcome.

Very difficult problem. Do you have many images to make a big training system and use deep learning? — hoaphumanoid, Feb 22 '16 at 08:08
@HoapHumanoid: yes I have terabytes of video. How can I use a DNN in this case? I've never used used it for image processing. — Alex, Feb 22 '16 at 09:25
take a look at a work like, e.g., [this one](http://www.robots.ox.ac.uk/~szheng/crfasrnndemo), or [this](http://www.cs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf) — Shai, Feb 22 '16 at 09:27
I think that a good start would be to use weak classifier, like Viola-Jones for faces, but train it for cows. If it does not work, then you can try deep learning. — FiReTiTi, Feb 22 '16 at 20:32
Two ideas: 1.) For brown cows, colour segmentation might be helpful. Won't work on cow colours which also show up in the background 2.) For a general solution, I'd rather look into motion detection instead of analyzing single images, using something like simply difference between two sequential images, or optical flow. If you detect a blob of similar motion direction of a certain size, this would be a cow if nothing else moves. A video sample sequence might be helpful to demonstrate a solution. — tfv, Apr 04 '16 at 19:26

score 1 · Answer 1 · answered Oct 25 '17 at 06:45

I realise this is an old thread, but I'd like to suggest an approach for problems like this one.

You can try with a texture based segmentation, as the grassy background has a different texture from the cow.

Take a look at this link where the texture energy features for an image are defined according to Laws' technique.

Here's an implementation of the laws technique in Python. It works by defining 2D kernels used to extract different features in an image, for instance edges, ripples, blobs and combinations thereof. The function below returns 9 images, from which texture features can be extracted.

def laws(array):

    # Define the 1D kernels
    L5 = np.array([1,4,6,4,1]) # level
    E5 = np.array([-1,-2,0,2,1]) # edge 
    S5 = np.array([-1,0,2,0,-1]) # spot
    R5 = np.array([1,-4,6,-4,1]) # ripples

    # Generate 2D kernels
    L5E5 = np.outer(L5,E5)
    E5L5 = np.outer(E5,L5)

    L5R5 = np.outer(L5,R5)
    R5L5 = np.outer(R5,L5)

    E5S5 = np.outer(E5,S5)
    S5E5 = np.outer(S5,E5)

    S5S5 = np.outer(S5,S5)

    R5R5 = np.outer(R5,R5)

    L5S5 = np.outer(L5,S5)
    S5L5 = np.outer(S5,L5)

    E5E5 = np.outer(E5,E5)

    E5R5 = np.outer(E5,R5)
    R5E5 = np.outer(R5,E5)

    S5R5 = np.outer(S5,R5)
    R5S5 = np.outer(R5,S5)


    return (0.5*(correlate(array, L5E5) + correlate(array, E5L5)), \
            0.5*(correlate(array, L5R5) + correlate(array, R5L5)), \
            0.5*(correlate(array, E5S5) + correlate(array, S5E5)), \
            correlate(array, S5S5), \
            correlate(array, R5R5), \
            0.5*(correlate(array, L5S5) + correlate(array, S5L5)), \
            correlate(array, E5E5), \
            0.5*(correlate(array, E5R5) + correlate(array, R5E5)), \
            0.5*(correlate(array, R5S5) + correlate(array, S5R5)))

score 0 · Answer 2 · answered Jun 09 '16 at 18:08

I would try taking two photos. One photo of a 'static' background with no cows, then a second photo with a cow(s). Then you can subtract those two images. I am not too familiar with python but imagemagick can difference images easily (http://www.imagemagick.org/Usage/compare/). Ideally the 'differenced' image will isolate the cow(s). From there you can get fancy and try various edge detection algorithms etc...

Hope that helps.

Challenging image segmentation: background and objects are similar

2 Answers2