Create a feature vector to classify segments in air images

Question

I am working on a project to segment air images and classify each segment. The images are very large and have huge homogeneous areas, so I decided to use a Split and Merge Algorithm for the segmentation.

(On the left the original image and on the right the segmented one, where each segment is represented in its RGB mean value Thanks to this answer)

For the classification I want to use a SVM Classifier (I used it a lot in two projects before) with a feature vector. For the beginning I just want to use five classes: Water, Vegetation, Built up area, Dune and Anomaly Now I am thinking about what I can put in this feature vector:

• The mean RGB Value of the Segment
• A texture feature (but can I represent the texture of the segment with just one value?)
• The place in the source image (maybe with a value which represents left, right or middle?)
• The size of the segment (Water segments should be much larger than built areas)
• The mean RGB values of the fourth neighborhood of the segment

So has anyone done something like this and can give me some advises what useful stuff I can put in the feature vector? And can someone give me an advise how I can represent the texture in the segment correctly?

Thank you for your help.

Answer 1

Instead of the Split and Merge algorithm, you could also use superpixels. There are several fast and easy-to-use superpixel algorithms available (some are even implemented in recent OpenCV versions). To name just a view:

• Compact Watershed (see here: https://www.tu-chemnitz.de/etit/proaut/forschung/rsrc/cws_pSLIC_ICPR.pdf)
• preSLIC and SLIC (see here: https://www.tu-chemnitz.de/etit/proaut/forschung/rsrc/cws_pSLIC_ICPR.pdf and here: http://www.kev-smith.com/papers/SLIC_Superpixels.pdf)
• SEEDS (see here: https://arxiv.org/abs/1309.3848)
• ERGC (see here: https://sites.google.com/site/pierrebuyssens/code/ergc)

Given the superpixel segmentation, there is a vast set of features you can compute in order to classify them:

• In Automatic Photo Pop-Up Table 1, Hoiem et al. consider, among others, the following features: mean RGB color, mean HSV color, color histograms, saturation histograms, Textons, differenty oriented Gaussian derivative filters, mean x and y location, area, ...
• In Recovering Occlusion Boundaries from a Single Image, Hoiem et al. consider some additional features to the above list in Table 1.
• In SuperParsing: Scalable Nonparametric Image Parsing with Superpixels , Tighe et al. additionally consider SIFT histograms, the mask reduced on a 8 x 8 image, boundign box shape, and color thumbnails.
• In Class Segmentation and Object Localization with Superpixel Neighborhoods , Fulkerson et al. also consider features from neighboring superpixels.

Based on the superpixels, you can still apply a simple merging-scheme in order to reduce the number of superpixels. Simple merging by color histograms might already be useful for your tasks. Otherwise you can additionally use edge information in between superpixels for merging.

Answer 2

You don't need to restrict yourself to just 1 feature vector. You could try multiple feature vectors (from the list you already have), and feed them to classifiers based on multiple kernel learning (MKL). MKL has shown to improve the performance over a single feature approach, and one of my favourite MKL techniques is VBpMKL.

If you have time, I would suggest you try one or more the following features, which can capture the features of interest:

• Haralick texture features
• Histogram oriented gradient features
• Gabor filters
• SIFT
• patch-wise RGB means