8

I am currently using OpenCV3.0 with the hope i will be able to create a program that does 3 things. First, finds faces within a live video feed. Secondly, extracts the locations of facial landmarks using ASM or AAM. Finally, uses a SVM to classify the facial expression on the persons face in the video.

I have done a fair amount of research into this but can't find anywhere the most suitable open source AAM or ASM library to complete this function. Also if possible I would like to be able to train the AAM or ASM to extract the specific face landmarks i require. For example, all the numbered points in the picture linked below: www.imgur.com/XnbCZXf

If there are any alternatives to what i have suggested to get the required functionality then feel free to suggest them to me.

Thanks in advance for any answers, all advice is welcome to help me along with this project.

EiiNHERJii1991
  • 91
  • 2
  • 3
  • 2
    [dlib](http://sourceforge.net/projects/dclib/files/dlib/) rules. you also can pass it cv::Mat and a Rect for te detection. – berak Apr 15 '15 at 18:37
  • @berak I wonder why such nice tool was not more popular, but then I saw it is in SF. – dashesy Aug 18 '15 at 16:23
  • @berak after more digging it seems it is on GitHub now (https://github.com/davisking/dlib) even though some links in their website is still pointing to SF, that is excellent. – dashesy Aug 18 '15 at 17:13
  • Hey @dashesy , Any chance you can send me your email? I'd like to discuss with you about Computer Vision (if possible). Thanks! – Roi Mulia May 24 '16 at 13:41

4 Answers4

7

In the comments, I see that you are opting to train your own face landmark detector using the dlib library. You had a few questions regarding what training set dlib used to generate their provided "shape_predictor_68_face_landmarks.dat" model.

Some pointers:

  • The author (Davis King) stated that he used the annotated images from the iBUG 300-W dataset. This dataset has a total of 11,167 images annotated with the 68-point convention. As a standard trick, he also mirrors each image to effectively double the training set size, ie 11,167*2=22334 images. Here's a link to the dataset: http://ibug.doc.ic.ac.uk/resources/facial-point-annotations/
    • Note: the iBUG 300-W dataset includes two datasets that are not freely/publicly available: XM2VTS, and FRGCv2. Unfortunately, these images make up a majority of the ibug 300-W (7310 images, or 65.5%).
    • The original paper only trained on the HELEN, AFW, and LFPW datasets. So, you ought to be able to generate a reasonably-good model on only the publicly-available images (HELEN,LFPW,AFW,IBUG), ie 3857 images.
      • If you Google "one millisecond face alignment kazemi", the paper (and project page) will be the top hits.

You can read more about the details of the training procedure by reading the comments section of this dlib blog post. In particular, he briefly discusses the parameters he chose for training: http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html

With the size of the training set in mind (thousands of images), I don't think you will get acceptable results with just a handful of images. Fortunately, there are many publicly available face datasets out there, including the dataset linked above :)

Hope that helps!

ekim
  • 71
  • 1
  • 3
  • I trained on 60 images using a reduced marker set - I'd like to think I was being conscientious of the relevant factors and the images were good, but regardless, the results were poor... Don't take much from the comment though, I don't know what I don't know about implementing the process. – Lamar Latrell Apr 04 '16 at 20:09
2

AAM and ASM are pretty old school and results are a little bit disappointing.

Most of Facial landmarks trackers use cascade of patches or deep-learning. You have DLib that performs pretty well (+BSD licence) with this demo, some other on github or a bunch of API as this one that is free to use.

You can also give a look at my project using C++/OpenCV/DLib with all functionalities you quoted and perfectly operational.

Tom A
  • 639
  • 5
  • 10
  • Thanks a lot for your detailed response, I feel like i know which direction i need to go now. thank you very much. – EiiNHERJii1991 Apr 15 '15 at 21:44
  • I've just downloaded and tested out your program and I am extremely impressed. I currently have my face detection running very well using OpenCV. Could you possibly point me towards what i need to do next using DLib to track my facial landmarks in video (similar to the mesh thats shown when model 1 is selected in your program). And most importantly how I could extract the co-ordinates of these values? thank you in advance, i appreciate any advice and help. – EiiNHERJii1991 Apr 15 '15 at 22:02
  • Check the doc and the demo but you need to convert opencv mat to dlib format then use face rect, image and model in dlib function to get facial landmarks. – Tom A Apr 15 '15 at 22:13
  • Sorry to bother you again, but i have started to track face landmarks in my own program from a live webcam feed using the "shape_predictor_68_face_landmarks.dat". However I was wondering if it is possible to change this to reflect more closely the few landmarks I would like to track in the image linked above? Or what i need to do to achieve these results. Thank you again for your time and help. – EiiNHERJii1991 Apr 15 '15 at 23:53
  • Don't worry I have found the examples of how to train such data files and will be trying it for myself soon. I will be using my own images containing faces for the training. In the example they use 4 images with several faces marked, and 5 more images with faces marked for testing. DO you happen to know how many images containing how many faces for training and testing is optimal? or should i closely mimic those used for the training of the original files? Don't worry if you either don't know or don't feel motivated to answer any longer, i have bugged you enough already :) – EiiNHERJii1991 Apr 16 '15 at 00:27
  • I never trained a model but as far as I know you need 100-1000 training images. Good luck :) – Tom A Apr 16 '15 at 09:22
1
  • Try Stasm4.0.0. It gives approximately 77 points on face.
Sagar Patel
  • 864
  • 1
  • 11
  • 22
0

I advise you to use FaceTracker library. It is written in C++ using OpenCV 2.x. You won't be disappointed on it.

Kornel
  • 5,264
  • 2
  • 21
  • 28