2

I have 2 issues when using PhotoCapture on the Hololens 2, which are probably connected. They did not occur on the Hololens 1.

  • The only obtainable resolution is 3904x2196
  • Getting the CameraToWorldMatrix always fails

As can be seen in the docs, this resolution has no framerate associated with it. My presumption is that the CameraToWorldMatrix is only obtainable for camera profiles with a lower resolution.

How can i change the resolution and get the matrix within Unity?

Minimal Reproducible Example

I am using Unity 2019.2.19f1 and Visual Studio 2019 Community (16.4.5)

  1. Create a new Unity Project following the steps here, except using IL2CPP as the scripting backend instead of .NET
  2. In Player Settings: Capabilities enable InternetClient, InternetClientServer, PrivateNetworkClientServer, Webcam, Microphone and SpatialPerception, in Player Settings: Supported Device Families select Holographic

  3. Create a new empty gameobject and add the following script:

    using System.Collections;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;
using UnityEngine.Windows.WebCam;

public class PhotoCaptureController : MonoBehaviour
{
    PhotoCapture photoCaptureObject = null;

    bool isRunning = false;

    void Start()
    {
        StartCoroutine(StartCameraCapture());
    }

    private IEnumerator StartCameraCapture()
    {
        if (!Application.HasUserAuthorization(UserAuthorization.WebCam))
        {
            yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
        }
        if (Application.HasUserAuthorization(UserAuthorization.WebCam))
        {
            Debug.Log("Creating PhotoCapture");
            PhotoCapture.CreateAsync(false, OnPhotoCaptureCreated);
        }
        else
        {
            Debug.Log("Webcam Permission not granted");
        }
    }

    private void Update()
    {
        if (isRunning)
        {
            photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
        }
    }

    void OnPhotoCaptureCreated(PhotoCapture captureObject)
    {
        photoCaptureObject = captureObject;

        IEnumerable<Resolution> availableResolutions = PhotoCapture.SupportedResolutions;

        foreach (var res in availableResolutions)
        {
            Debug.Log("PhotoCapture Resolution: " + res.width + "x" + res.height);
        }

        Resolution cameraResolution = availableResolutions.OrderByDescending((res) => res.width * res.height).First();

        CameraParameters c = new CameraParameters();
        c.hologramOpacity = 0.0f;
        c.cameraResolutionWidth = cameraResolution.width;
        c.cameraResolutionHeight = cameraResolution.height;
        c.pixelFormat = CapturePixelFormat.BGRA32;

        captureObject.StartPhotoModeAsync(c, OnPhotoModeStarted);
    }

    private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
    {
        if (result.success)
        {
            isRunning = true;
            photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
        }
    }

    void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame frame)
    {
        if (result.success)
        {
            if (frame.TryGetCameraToWorldMatrix(out Matrix4x4 cameraToWorldMatrix))
            {
                Debug.Log("Successfully obtained CameraToWorldMatrix: " + cameraToWorldMatrix.ToString());

            }
            else
            {
                Debug.Log("Failed to obtain CameraToWorldMatrix");
            }
        }
        frame.Dispose();
    }
}
  4. Change Build Settings:enter image description here
  5. Build, open VS solution, set build target to ARM64, Debug and deploy to Device
Commodore Yournero
  • 383
  • 3
  • 13
  • Could you add the code you are using? – derHugo Feb 26 '20 at 15:02
  • @derHugo its basically copied from [Unity Docs](https://docs.unity3d.com/ScriptReference/Windows.WebCam.PhotoCapture.html) and [MS Docs](https://learn.microsoft.com/en-us/windows/mixed-reality/locatable-camera-in-unity) and adapted a bit – Commodore Yournero Feb 28 '20 at 11:54
  • What version of unity are you using? Did an error occur at any step? At the moment, it's difficult for us to find the cause of the problem from your current post. We recommend that you provide an MVCE(https://stackoverflow.com/help/minimal-reproducible-example) and the steps to reproduce the issue, so that we can locate the problem or find a solution. – Hernando - MSFT Mar 02 '20 at 11:58
  • @Hernando-MSFT do you know of anyone who did NOT encounter those 2 issues? I could provide code, but i really doubt that the issue is on my side since the same code worked as expected on the Hololens 1 – Commodore Yournero Mar 02 '20 at 12:21
  • @Hernando-MSFT issue occurs with Unity 2019.2.19 and 2018.4.17 (LTS) – Commodore Yournero Mar 02 '20 at 16:23
  • There is no other feedback on this issue at this time. We are creating the sample following the Unity and MS documentation to reproduce this issue. If you can provide MVCE that will save time and help us locate issue faster. – Hernando - MSFT Mar 05 '20 at 07:10
  • @Hernando-MSFT Edited the question to include a MVCE. If you are investigating this, i can also add my ouput logs on app startup – Commodore Yournero Mar 05 '20 at 10:37

1 Answers1

1

This CameraCapture plugin may work for you. It's a native plugin for Unity. The readme.txt contains information on building it. Basically, you need to build the native plugin variants before trying to open up the Unity sample. If you haven’t built C++/WinRT components, you may need to go through the setup steps here. The SpatialCameraTracker.cs file shows how to receive the camera matrices.

There is minimal support on this but all the code is in the repo

Kevin K
  • 11
  • 2