What are good features for classifying photos of clothing?

Question

I want to build a clothing classifier that takes a photo of an item of clothing and classifies it as 'jeans', 'dress', 'trainers' etc.

Some examples:

jeans trainer enter image description here

These images are from retailer websites, so are typically taken from the same angle, typically on a white or pale background -- they tend to be very similar.

I have a set of several thousand images whose category I already know, which I can use to train a machine-learning algorithm.

However, I'm struggling for ideas of what features I should use. The features I have so far:

def get_aspect_ratio(pil_image):
    _, _, width, height = pil_image.getbbox()

    return width / height


def get_greyscale_array(pil_image):
    """Convert the image to a 13x13 square grayscale image, and return a
    list of colour values 0-255.

    I've chosen 13x13 as it's very small but still allows you to
    distinguish the gap between legs on jeans in my testing.

    """
    grayscale_image = pil_image.convert('L')
    small_image = grayscale_image.resize((13, 13), Image.ANTIALIAS)

    pixels = []
    for y in range(13):
        for x in range(13):
            pixels.append(small_image.getpixel((x, y)))

    return pixels


def get_image_features(image_path):
    image = Image.open(open(image_path, 'rb'))

    features = {}
    features['aspect_ratio'] = get_aspect_ratio(image)

    for index, pixel in enumerate(get_greyscale_array(image)):
        features["pixel%s" % index] = pixel

    return features

I'm extracting a simple 13x13 grayscale grid as a crude approximation of shape. Howerver, using these features with nltk's NaiveBayesClassifier only gets me 34% accuracy.

What features would work well here?

if you use a bigger image than 13x13 do you get better accuracy? — Joran Beasley, Sep 19 '13 at 16:27
@JoranBeasley moving to a 20x20 image actually hurts accuracy by 2% and significantly hurts performance. — Wilfred Hughes, Sep 19 '13 at 16:34
what if you simplify it to "shoe or not shoe" and start with that ... maybe change some of your classifications (I dont know if you manually setup classifications, not used bltk) — Joran Beasley, Sep 19 '13 at 16:43
To the people who marked this question as too broad: do you think there are changes possible to make it less broad, or is it unfixable? I am talking about a specific dataset and have received some excellent answers. — Wilfred Hughes, Sep 20 '13 at 11:06
This article proposes SVM with PCA: https://medium.com/@dimart/pok%C3%A9mon-recognition-d3ad5cadc61e — Wilfred Hughes, Dec 16 '15 at 11:28

score 10 · Accepted Answer · answered Sep 19 '13 at 17:16

This is a tough problem and therefore there are many approaches.

On common method (although complicated) is taken an input image, superpixelate the image and compute descriptors (such as SIFT of SURF) of those superpixels building a bag-of-word representation by accumulating histograms per superpixel, this operation extracts the key information from a bunch of pixels reducing dimensionality. Then a Conditional Random Field algorithm searches for relationships between superpixels in the image and classifies the group of pixels inside a known category. For pixelating images scikit-image package implements SLIC algorithm segmentation.slic, and for the CRF you should take a look to PyStruct package. SURF and SIFT can be calculated using OpenCV.

enter image description here

Another simple version would be computing descriptors of a given image (SIFT, SURF, borders, histogram etc) and use them as inputs in a classifier algorithm, you might want start from here, maybe scikit-learn.org is the easiest and most powerful package for doing this.

score 2 · Answer 2 · answered Sep 19 '13 at 18:05

HOG is commonly used in object detection schemes. OpenCV has a package for HOG descriptor:

http://docs.opencv.org/modules/gpu/doc/object_detection.html

You can also use BoW based features. Here's a post that explains the method: http://gilscvblog.wordpress.com/2013/08/23/bag-of-words-models-for-visual-categorization/

user2683129 · Answer 3 · 2013-09-19T23:04:44.930

Using all the raw pixel values in an image directly as features aren't great, especially as the number of features increase, due to the very large search space (169 features represents a large search space, which can be difficult for any classification algorithm to solve). This is perhaps why moving to an 20x20 image actually degrades performance compared to 13x13. Reducing your feature set/search space might improve performance since you simplify the classification problem.

A very simple (and generic) approach to accomplish this is to use pixel statistics as features. This is the mean and standard deviation (SD) of the raw pixel values in a given region of the image. This captures the contrast/ brightness of a given region.

You can choose the regions based on trial and error, e.g., these can be:

a series of concentric circular regions, increasing in radius, in the centre of the image. The mean and SD of four circular regions of increasing size gives eight features.
a series of rectangular regions, either increasing in size or fixed size but placed around different regions in the image. The mean and SD of four non-overlapping regions (of size 6x6) in the four corners of the image and one in the centre gives 10 features.
a combination of circular and square regions.

score 0 · Answer 4 · answered Sep 19 '13 at 17:11

0

Have you tried SVM? It generally performs better than Naive Bayes.

answered Sep 19 '13 at 17:11

Beibei

586
1
4
6

What are good features for classifying photos of clothing?

4 Answers4

Linked