0

I am developing a Hololens application that requires me to determine the user's eye position from the front-facing camera's captured image. This information is crucial in identifying the real-world object that the user is looking at. I have tried two existing methods but encountered some challenges:

  • First Method: I extracted GazeOrigin and GazeDirection from the camera's image and calculated the intersection point of that ray with the viewport. However, I encountered problems due to differences in intrinsic and extrinsic properties between the front-facing camera and the FOV (field of view) camera.

  • Second Method: I calculated the 3D position of the intersection point between GazeDirection and SpatialMapping mesh and projected this point onto the front-facing camera and FOV. This method does not work well with dynamic objects such as humans or books held in front of the user.

I am looking for alternative solutions that can handle dynamic objects or improvements to the above methods that can effectively solve my problem. Any insights or suggestions would be appreciated.

user22087954
  • 1
  • Have you tried setting the Gaze Point to be visible? This way the Gaze Point will be captured together when the image is captured. – Zuocheng Wang - MSFT Jun 20 '23 at 08:47
  • Yes, I have. But there is 2 problem with this method which is it does not work well with dynamic objects such as humans or books held in front of the user and i can't get the (x, y) position on the image. – user22087954 Jun 21 '23 at 11:17
  • Setting the Gaze Point to be visible is just to know which object the user is looking at, not to get a precise position. For Spatial Mapping in HoloLens, it is designed to scan rooms, not for dynamic objects. For your needs, you may need to identify the object first and then get its position. You can try QR Code or some third-party libraries for identifying objects. – Zuocheng Wang - MSFT Jun 28 '23 at 09:15

0 Answers0