0

I am trying to create an AR debugging application that takes information from a server (the phone) and displays it on a client (the computer).

I am using the AROcclusionManager component to get environmentDepthImage information. I am using a native method (TryAcquireEnvironmentDepthCpuImage) that returns an XRCpuImage with information relating to the environmentDepthImage. I'm diving into the (image)Plane property of the XRCpuImage and am extracting the byteArray that is a property of XRCpuImage through XRCpuImage.GetPlane(); . I am putting this information into a custom serializable class and transferring it as metadata through klakNdi. on the other side, once I deserialize the byteArray, it is populated with information, but I am having trouble getting this byte array into a format that I can display in the unity editor. It is showing as DepthUint16 format server side, and it is also showing the R16 format supported, but once I get the information on the other side, I can't put the byte array into any format I can display in the editor.

this is my Class constructor:

    public SerializableDepthImage(XRCpuImage depthImage)
    {
        width = depthImage.width;
        height = depthImage.height;
        format = depthImage.format;
        planeCount = depthImage.planeCount;
        texData = new byte[depthImage.GetPlane(0).data.Length];
        depthImage.GetPlane(0).data.CopyTo(texData);
        depthImage.Dispose();
    }

this is the code where I'm deserializing the SerializableDepthImage:

if (remotePacket.depthImage.texData == null)
{
    Debug.Log("texData is null");
}
else
{
    Debug.Log("texData is not null");
    
    Texture2D newDepthImage = remotePacket.depthImage.
                              ReconstructTexture2DFromSerializableDepthImage(remotePacket.depthImage);
    Debug.Log("serialized depthImage texData byte value: " +
               remotePacket.depthImage.texData[remotePacket.depthImage.texData.Length -100]);
    Debug.Log("newDepthImage width = " + newDepthImage.width + 
              "newDepthImage.height = " + newDepthImage.height + 
              "newDepthImage.format = " + newDepthImage.format);
    
    depthImage.texture = newDepthImage;

but after this assignation, nothing shows in unity. I'm at my wits end... any help would be greatly appreciated!

I was expecting the depthImage.texture = newDepthImage code to correctly assign the Texture2D newDepthImage to the texture property of the RawImage depthImage. It is just showing as black.

J Michael Sielaff
  • 1
  • 1
  • Your serialization method most likely gets an array containing raw pixel values. When it is reconstructed you need to check that the input should also be raw pixel values, and not a byte array representing an *image file*. As a rule, when handling raw values you need to also pass around width/height/stride/pixelFormat. If these things are not required, the method likely expects a file, where all of these things are built in. – JonasH Aug 25 '23 at 07:51

0 Answers0