SpatialCoordinateSystem.TryGetTransformTo() from Webcam to Unity space fails in a non-initial, separated spatial environment on HoloLens 2

Question

I have a quite specific problem regarding to a transformation matrix for transformations from the HoloLens 2 webcam space into the current Unity scene space in a Unity+MRTK+OpenXR app. The goal is to acquire the exact camera pose related to a camera frame, which was acquired through Windows.Media.Capture, in Unity space.

My environment:

• Unity 2021.3.8.
• MRTK v2.8.2
• Mixed Reality OpenXR Plug-In v1.6.0

For obtaining the matrix, I first receive a Windows.Perception.Spatial.SpatialCoordinateSystem instance (unityReferenceCoordinateSystem) representing the Unity Space through the MR OpenXR Plug-In as described HERE:

using Windows.Perception.Spatial;
using Microsoft.MixedReality.OpenXR;

SpatialCoordinateSystem unityReferenceCoordinateSystem = PerceptionInterop.GetSceneCoordinateSystem(Pose.identity) as SpatialCoordinateSystem;

and I obtain the camera space (cameraCoordinateSystem) from the Windows.Media.Capture.Frames.MediaFrameReference camera frame instance acquired from a MediaFrameReader by

MediaFrameReference mediaFrame; // acquired camera frame
SpatialCoordinateSystem cameraCoordinateSystem = mediaFrame.CoordinateSystem;

Finally I obtain the required transformation matrix by using SpatialCoordinateSystem.TryGetTransformTo() as you can see in my complete method:

using Microsoft.MixedReality.Toolkit;

public bool TryGetCameraToUnityMatrix(out Matrix4x4 cameraToUnity)
{
    // (obtain MediaFrameReader, acquire a camera frame and obtain 
    // unityReferenceCoordinateSystem and cameraCoordinateSystem as described above)
    
    System.Numerics.Matrix4x4? camToUnitySysMatrix = cameraCoordinateSystem.TryGetTransformTo(unityReferenceCoordinateSystem);

    if (!camToUnitySysMatrix.HasValue)
    {
        return false;
    }

    cameraToUnity = camToUnitySysMatrix.Value.ToUnity();
    return true;
}

This works all fine so far - until I bring the HoloLens into another spatial environment, which is not connected to the environment, which was present when the app was started.

Describing the following scenario should make clear what I mean by that:

1. Start the app on HL2
2. Acquire the cameraToUnity matrix as described --> works fine
3. Set the HL to stand-by
4. Go to another room, for which the HL's spatial awareness does not know the connection between these two rooms
5. Wake up HL and open the (still running) app.
6. Acquire the cameraToUnity matrix. --> FAILS:
   camToUnitySysMatrix.HasValue returns false (even though both arguments unityReferenceCoordinateSystem and cameraCoordinateSystem are not null.)
7. Set the HL to stand-by again
8. Go back to the initial environment where the app was originaly started
9. Wake up HL and open the (still running) app.
10. Acquire the cameraToUnity matrix as described --> works fine again! (camToUnitySysMatrix has valid value again)

I also made sure that unityReferenceCoordinateSystem = PerceptionInterop.GetSceneCoordinateSystem(Pose.identity) is re-called after I changed the environment and also the MediaFrameReader gets freshly instantiated origining from a new MediaCapture instance.

But obviously a transformation between the two SpatialCoordinateSystems seems to fail if it is attempted in the non-initial spatial environment.

Any ideas on how to solve this?

UPDATE

A minimal Unity sample project for reproducing this problem can be found here: https://github.com/pjaydev/trygettransformto-so

Answer 1

Usually, users should wear the HoloLens device and keep it turned on while moving and operating, so that the device can fully understand the environment. And HoloLens also has some requirements on the size of the site used, if the room is too small, HoloLens may not work properly. If you have confusion about this issue or there is business impact, you can submit a ticket via aka.ms/HLSupport.