3D SIFT for human activity classification in videos. NOT GETTING GOOD ACCURACY.

Question

I am trying to classify human activities in videos(six classes and almost 100 videos per class, 6*100=600 videos). I am using 3D SIFT(both xy and t scale=1) from UCF.

for f= 1:20
f
offset = 0;
c=strcat('running',num2str(f),'.mat');
load(c)
pix=video3Dm;
% Generate  descriptors at locations given by subs matrix
for i=1:100
 reRun = 1;
while reRun == 1
    loc = subs(i+offset,:);

    fprintf(1,'Calculating keypoint at location (%d, %d, %d)\n',loc);

    % Create a 3DSIFT descriptor at the given location
    [keys{i} reRun] = Create_Descriptor(pix,1,1,loc(1),loc(2),loc(3));

    if reRun == 1
        offset = offset + 1;
    end

     end
  end
  fprintf(1,'\nFinished...\n%d points thrown out do to poor descriptive        ability.\n',offset);

for t1=1:20
des(t1+((f-1)*100),:)=keys{1,t1}.ivec;
end
f
end

My approach is to first get 50 descriptors(of 640 dimension) for one video, and then perform bag of words with all descriptors(on 50*600= 30000 descriptors). After performing Kmeans(with 1000 k value)

 idx1000=kmeans(double(total_des),1000);

I am getting 30k of length index vector. Then I am creating histogram signature of each video based on their index values in clusters. Then perform svmtrain(sum in matlab) on signetures(dim-600*1000).
Some potential problems-

1-I am generating random 300 points in 3D to calculate 50 descriptors on any 50 points from those points 300 points.

2- xy, and time scale values, by default they are "1".

3-Cluster numbers, I am not sure that k=1000 is enough for 30000x640 data.

4-svmtrain, I am using this matlab library.

NOTE: Everything is on MATLAB.

Answer 1

Your basic setup seems correct especially given that you are getting 85-95% accuracy. Now, it's just a matter of tuning your procedure. Unfortunately, there is no way to do this other than testing a variety of parameters examining the results and repeating. I going to break this answer into two parts. Advice about bag of words features, and advice about SVM classifiers.

Tuning Bag of Words Features

You are using 50 3D SIFT Features per video from randomly selected points with a vocabulary of 1000 visual words. As you've already mentioned, the size of the vocabulary is one parameter you can adjust. So is the number of descriptors per video.

Let's say that each video is 60 frames long, (at 30 fps only 2 sec, but let's assume you are sampling at 1fps for a 1 minute video). That means you are capturing less than one descriptor per frame. That seems very low to me even with 3D descriptors especially if the locations are randomly chosen.

I would manually examine the points for which you are generating features. Do they appear be well distributed in both space and time? Are you capturing too much background? Ask yourself, would I be able to distinguish between actions given these features?

If you find that many of the selected points are uninformative, increasing the number of points may help. The kmeans clustering can make a few groups for uninformative outliers, and more points means you hopefully capture a few more informative points. You can also try other methods for selecting points. For example, you could use corner points.

You can also manually examine the points that are clustered together. What sorts of structures do the groups have in common? Are the clusters too mixed? That's usually a sign that you need a larger vocabulary.

Tuning SVMs

Using the Matlab SVM implementation or the Libsvm implementation should not make a difference. They are both the same method and have similar tuning options.

First off, you should really be using cross-validation to tune the SVM to avoid overfitting on your test set.

The most powerful parameter for the SVM is the kernel choice. In Matlab, there are five built in kernel options, and you can also define your own. The kernels also have parameters of their own. For example, the gaussian kernel has a scaling factor, sigma. Typically, you start off with a simple kernel and compare to more complex kernels. For example, start with linear, then test quadratic, cubic and gaussian. To compare, you can simply look at your mean cross-validation accuracy.

At this point, the last option is to look at individual instances that are misclassified and try to identify reasons that they may be more difficult than others. Are there commonalities such as occlusion? Also look directly at the visual words that were selected for these instances. You may find something you overlooked when you were tuning your features.

Good luck!