1

I am getting facial landmarks using dlib. I have dataset of more than 1000 faces. I want to make a comparison of these 1000 images with some unknown image. To decrease the database search time, i want to cluster these 1000images in to 10 different clusters based on the 68 facial landmark features of dlib. Currently, I am clustering based on chin to nose distance of different face images.

Problem: Each image of same person is generating different facial landmarks which is effecting the distance calculated from chin to nose tip. Please find the screenshot of csv enter image description here

  1. 1st column - Face Image names (same person face with around 25 samples)
  2. 2nd,3rd columns - Kmeans clustered labels and centroids of column 4
  3. 4th - Face chin to nose tip euclidean distances
  4. 5th - 68 long dlib facial landmarks seperated as chin, eye ....

Questions:

  1. Is it a right way to cluster images based on facial landmarks ? If not, what is the best way to cluster face images/face groupping to make databse searc more efficient for more images ?

I tried with gender classification, but the accuracy is not good. Tried with face color/Ethnicity classification but this limiting my scope. For instance, only asian/european faces will again make me to search all the database

I am not able to identify which is the right factor to cluster. Any reference to articles or ideas are much appreciated.

user1
  • 391
  • 3
  • 27
  • 1
    If clustering is what you want: Consider other approaches like FaceNet (face -> euclidean-space embedding; at least one very popular python-based implementation: [openface](https://github.com/cmusatyalab/openface) which even has this example). For query stuff you probably should more look into other directions, learning to hash and co. Of if you are in metric-space (facenet): approximate nearest-neighbor search. – sascha Feb 03 '18 at 14:57
  • 2
    Why don't you use the face clustering example program that comes with dlib? http://blog.dlib.net/2017/02/high-quality-face-recognition-with-deep.html. I assume that's where you saw dlib's chinese whispers code in the first place? – Davis King Feb 03 '18 at 15:57
  • Thank you @sascha, yes, it is much more appropriate. – user1 Feb 03 '18 at 16:36
  • Thanks @Davis King, I'll check it. – user1 Feb 03 '18 at 16:37
  • @sascha, Could you please explain me what you meant by "For query stuff .....learning to hash and co". I am sorry, i didn't get you. – user1 Feb 03 '18 at 16:58
  • @krishnadamarla Something like [that](https://pdfs.semanticscholar.org/595d/0fe1c259c02069075d8c687210211908c3ed.pdf) (academic survey pdf link) – sascha Feb 03 '18 at 18:05
  • Got it. Thanks @sascha – user1 Feb 05 '18 at 13:52
  • @DavisKing, is python version of chineese whispers available for dlib ? i am working with python and i see dlib has only c++ support for this. Can we do incremental learning with this api http://dlib.net/dlib/clustering/chinese_whispers_abstract.h.html – user1 Feb 19 '18 at 09:50
  • 1
    Yes: http://dlib.net/face_clustering.py.html – Davis King Feb 19 '18 at 13:00
  • Thanks @DavisKing, could you please tell me how i can add new faces to http://dlib.net/face_clustering.py.html for incremental learning ? – user1 Feb 19 '18 at 14:10

1 Answers1

1

Clustering in the way i specified in question is not correct for facial images. It is best to use a convolutional neural network to train the features instead of manually computing distances from facial landmarks.

Later on these trained features, we can apply any of the popular clustering algorithm as shown here: https://arxiv.org/pdf/1604.00989.pdf or as @sascha suggested, Approximate Nearest Neighbour or as @Davis King suggested Chinese Whispers depending on your needs.

As @sascha suggested, there are many deep learning libraries like openface that does this on top of torche/tensorflow.

user1
  • 391
  • 3
  • 27
  • I am not quite sure whether it is a classification/clustering problem though. @DavisKing, used chineese whispers clustering in his dlib libraray. Why ? Why not nearest neighbour which is tree based ? One reason could be to use graph based algorithm for fastness. Any other reasons ? Mabe it depends on our applications ? – user1 Apr 06 '18 at 09:11