-2

My question is how can I retrieve and store the eye-tracking/eye gaze data (such as the coordinate position of where the user is looking at different timestamps or the heatmap) with HoloLens 2 and MRTK in Unity?

I have seen some C# codes for getting the direction and origin of the gaze but I have no idea where to put them or where/how to add them (and to what).

Is it possible to retrieve and store such data without doing coding and directly from within the Unity and MRTK? If no, then how can I do that with coding? and where to find the retrieved data?

Appreciate your help.

Arde
  • 41
  • 1
  • 6

1 Answers1

0

To access eye tracking data, simply use CoreServices.InputSystem.EyeGazeProvider in your MonoBehaviour script.

The GazeOrigin property of EyeGazeProvider will return gaze ray origin and GazeDirection property will return gaze ray direction. More infromation please see: Accessing eye tracking data in your Unity script

Hernando - MSFT
  • 2,895
  • 1
  • 5
  • 11
  • Thanks for the response Hernando. I'm using MRTK in Unity which already has these scripts and they work perfectly. My question is how to retrieve the eye-tracking data such as where the user is looking at in form of coordinate points (x,y,z) at different timestamps. I need to have these points as some form of a dataset. How and where these coordinates (or even the gaze origin or direction data) are generated and stored? Any thoughts? – Arde Jul 25 '20 at 23:58
  • The eye gaze coordinate point is provided in Vector3 in `HitPosition` field. As shown in this sample:[Examples for using CoreServices.InputSystem.EyeGazeProvider](https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/EyeTracking/EyeTracking_EyeGazeProvider.html#examples-for-using-coreservicesinputsystemeyegazeprovider). And as the answer you have referred, MRTK does not provide a method to return the position with a timestamp but there's nothing preventing you from doing it in your own code. – Hernando - MSFT Jul 27 '20 at 08:16
  • Thanks Hernando, but how can I get the values of HitPosition as output? When I run the eye-tracking app in HoloLens, I see that everything is working fine, but I do not see any generated outputs like coordinates. Could you elaborate more? – Arde Jul 27 '20 at 14:33
  • @Arde Actually, the gaze usually likes the `half-open line segment`, it starts at your eye, in one direction only along the line and proceeding indefinitely. And once the gaze hits an object, it will become a closed line segment and get both endpoints. At this time, the position field of another endpoint can be returned. So, to get the point coordinate you should hit an object first. – Hernando - MSFT Jul 28 '20 at 07:39
  • Dear Hernando, that is a great point! but I receive no outputs even when I hit an object with eye gaze (e.g. target positioning example). So my question is: are the coordinate points automatically recorded and stored somewhere in the device or in a log file when hitting an object? If yes, where can I find that output? if no, how can I generate that output? You already mentioned that the HitPosition field includes those outputs, but how can I store or print this field to see the actual coordinates or points? – Arde Jul 28 '20 at 18:31
  • The following code can log coordinates in the Console windows: `Debug.Log(CoreServices.InputSystem.EyeGazeProvider.HitPosition);` You can programmatically store this data in anywhere. If you have any other question, please let me know. – Hernando - MSFT Jul 29 '20 at 08:55
  • Great! I am seeing several scripts in the MRTK Foundation and Example packages for eye-gaze (e.g. "IMixedRealityEyeGazeDataProvider", "IMixedRealityEyeGazeProvider", "FollowEyeGaze", "FollowEyeGazeGazeProvider", etc). Could you tell me where (which script) I should add this line of code? – Arde Jul 29 '20 at 15:23
  • You should create a new script and attach it to a game object, and then add this line of code in the `Update` function of this new script. For this you need some basic knowledge about Unity, It is recommended that reading the official [Unity documentation](https://unity.com/learn) first. – Hernando - MSFT Jul 30 '20 at 08:55
  • Thanks Hernando! I used the following code for the new script: using System.Collections; using System.Collections.Generic; using UnityEngine; namespace Microsoft.MixedReality.Toolkit.Examples.Demos.EyeTracking { public class PrintLog : MonoBehaviour { void Update() { Debug.Log(CoreServices.InputSystem.EyeGazeProvider.HitPosition); } } } It worked in Unity (got the output in console). But do you also know how I can do the same when I deploy the app in HoloLens? I did not see any coordinate output in HoloLens or Visual Studio. – Arde Jul 31 '20 at 15:40