0

Can anyone point me in the direction of a library that can estimate face pose from 68 landmark points in Python?

To be more specific, many such libraries exist. However, they usually start from an image. I want to skip the landmark detection because I already have it. I just want to feed the 68-point landmarks to a function and retrieve the rotation and translation of the face.

If I wasn't a complete Python newbie, I may be able tp extract the relevant bits and pieces from existing libraries but I am finding this hard to do.

Thanks.

djip.co
  • 997
  • 10
  • 17

1 Answers1

1

This library seems to get your job done https://github.com/yinguobing/head-pose-estimation

It seems to work in 3 steps,

  1. Face Detection
  2. Landmark Detection
  3. Pose Estimation. After getting 68 facial landmarks, the pose could be calculated by a mutual PnP algorithm.

So, you should look through the codebase for this mutual PnP algorithm that is taking the 68 landmarks as an input.

Looks like all the code regarding pose estimation that you will need is inside this file!

To briefly summarize the whole file of pose estimation code for you

  • the PoseEstimator class contains all the relevant functions
  • the __init__ function is establishing self.model_points as the nose tip, chin, mouth coordinates. Its also handling camera internals, rotation and translation vectors. All of these are essential inputs to the PnP algorithm that is doing pose estimation
  • self._get_full_model_points() is reading the landmark points from a file and storing it in self.model_points_68
  • this class has the important function called solve_pose_by_68_points which is doing what you need. It is expecting an input of your landmarks.
  • this solve_pose_by_68_points() function is used inside this file
  • the input to that function is a variable called marks

You need to figure out a way to feed your own landmark points into this solve_pose_by_68_points function, using the marks variable as a guide.

Or, just recreate the code from the pose_estimator.py file and the solve_pose_by_68_points function within it.

Side note - that function is simply calling cv2.solvePnP, openCVs implementation of the PnP algorithm I'm guessing.

ranvit
  • 99
  • 6
  • Thank you so much for your pointers! It does looklike the solution is in this file. I will give it a try and report back. – djip.co May 05 '21 at 03:45
  • I made some progress but there are choke points. When I call solve_pose_by_68_points(), I need to supply an OpenCV Mat object. What I have is a 2-dimension array. Currently, I'm using this code to convert my points array to a Numpy array which, from what I read elsewhere, is supposed to be the same as a Mat object in Python: numpy.array(points, np.float32) I have no clue if this is the proper way. I tried using the rotation data and the numbers are so small that I can't notice an actual pose change... – djip.co May 11 '21 at 20:42
  • share your code somewhere, help me run it. give me your points array, and give me the solve_pose_by_68_points() function – ranvit May 17 '21 at 05:32
  • Thanks for getting back to me. I made a gist where I put what I have so far. I do get numbers at the end but I do not think they are right... I'm not sure how to create a cv.Mat from an array. Here's the gist: https://gist.github.com/djipco/30d0108cc19b04fd8acd35881ff7defc – djip.co May 17 '21 at 13:51