Best Facial Landmark that can be easily extracted from NIR Image?

Question

I'm playing with Eye Gaze estimation using a IR Camera. So far i have detected the two Pupil Center points as follows:

1. Detect the Face by using Haar Face cascade & Set the ROI to Face.
2. Detect the Eyes by using Haar Eye cascade & Name it as Left & Right Eye Respectively.
3. Detect the Pupil Center by thresholding the Eye region & found the pupil center.

So far I've tried to find the gaze direction by using the Haar Eye Boundary region. But this Haar Eye rect is not always showing the Eye Corner points. So the results was poor.

Then I've tried to tried to detect Eye Corner points using GFTT, Harriscorners & FAST but since I'm using NIR Camera the Eye Corner points are not clearly visible & so i cant able to get the exact corner positions.So I'm stuck here!

What else is the best feature that can be tracked easily from face? I heard about Flandmark but i think that is also will not work in IR captured images.

Is there any feature that can be extracted easily from the face images? Here I've attached my sample output image.

Answer 1

I would suggest flandmark, even if your intuition is the opposite - I've used it in my master thesis (which was about head pose estimation, a related topic). And if the question is whether it will work with the example image you've provided, I think it might detect features properly - even on a gray scaled image. I think in the flandmark they probably convert to image to grayscale before applying a detector (like the haar detector works). Moreover, It surprisingly works with the low resolution images what is an advantage too (especially when you're saying eye corners are not clearly visible). And flandmark can detect both eye corners, mouth corners and nose tip (actually I will not rely on the last one, from my experience detecting nose tip on the single image is quite noisy, however works fine with an image sequence with some filtering e.g. average, Kalman). If you decide to use that technique and it works, please let us know!