Capturing a snapshot of VideoBackground within Unity Vuforia plugin

Question

I would like to capture the realworld view at the moment when content is placed using a GroundPlane or Mid Air stage.

This seems to be readily available within the AR camera's BackgroundPlane Mesh Renderer - Custom/VideoBackground (see screenshot below). However, when I try to access this texture and encode to JPG, the output image is black.

Here is the code I am testing with:

 MeshRenderer backgroundMesh=GameObject.Find("BackgroundPlane").GetComponent<MeshRenderer>();
      Texture2D texture=(Texture2D)backgroundMesh.material.mainTexture;

      byte[] bytes = texture.EncodeToJPG();
      var dirPath = Application.dataPath + "/../SavedImages/";
      if(!Directory.Exists(dirPath)) {
          Directory.CreateDirectory(dirPath);
      }
      File.WriteAllBytes(dirPath + "Image" + ".jpg", bytes);

Here is a screenshot of the vuforia settings for Video Background:

is the video background enabled in the VuforiaConfiguration (Resources)? — derHugo, Sep 06 '19 at 12:42

score 2 · Answer 1 · answered Feb 11 '20 at 17:15

You can use Vuforia image class for capturing the real world only.

The scripts are tested on mobile, and used in FMETP STREAM. For your case, you can convert texture2d as jpg.

using UnityEngine;
using System.Collections;
using Vuforia;
using UnityEngine.UI;

public class VuforiaCamAccess : MonoBehaviour
{
    private bool mAccessCameraImage = true;
    public RawImage rawImage;
    public GameObject Mesh;
    private Texture2D texture;

#if UNITY_EDITOR
    private Vuforia.PIXEL_FORMAT mPixelFormat = Vuforia.PIXEL_FORMAT.GRAYSCALE;
#else
    private Vuforia.PIXEL_FORMAT mPixelFormat =  Vuforia.PIXEL_FORMAT.RGB888;
#endif

    private bool mFormatRegistered = false;

    void Start()
    {
#if UNITY_EDITOR
        texture = new Texture2D(Screen.width, Screen.height, TextureFormat.R8, false);
#else
        texture = new Texture2D(Screen.width, Screen.height, TextureFormat.RGB24, false);
#endif
        // Register Vuforia life-cycle callbacks:
        Vuforia.VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted);
        Vuforia.VuforiaARController.Instance.RegisterOnPauseCallback(OnPause);
        Vuforia.VuforiaARController.Instance.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
    }

    private void OnVuforiaStarted()
    {
        // Try register camera image format
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
        {
            Debug.Log("Successfully registered pixel format " + mPixelFormat.ToString());
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register pixel format " + mPixelFormat.ToString() +
                "\n the format may be unsupported by your device;" +
                "\n consider using a different pixel format.");
            mFormatRegistered = false;
        }
    }

    private void OnPause(bool paused)
    {
        if (paused)
        {
            Debug.Log("App was paused");
            UnregisterFormat();
        }
        else
        {
            Debug.Log("App was resumed");
            RegisterFormat();
        }
    }

    private void OnTrackablesUpdated()
    {
        //skip if still loading image to texture2d
        if (LoadingTexture) return;

        if (mFormatRegistered)
        {
            if (mAccessCameraImage)
            {
                Vuforia.Image image = CameraDevice.Instance.GetCameraImage(mPixelFormat);
                //if (image != null && image.IsValid())
                if (image != null)
                {
                    byte[] pixels = image.Pixels;
                    int width = image.Width;
                    int height = image.Height;
                    StartCoroutine(SetTexture(pixels, width, height));
                }
            }
        }
    }

    bool LoadingTexture = false;
    IEnumerator SetTexture(byte[] pixels, int width, int height)
    {
        if (!LoadingTexture)
        {
            LoadingTexture = true;
            if (pixels != null && pixels.Length > 0)
            {
                if (texture.width != width || texture.height != height)
                {
#if UNITY_EDITOR
                    texture = new Texture2D(width, height, TextureFormat.R8, false);
#else
                    texture = new Texture2D(width, height, TextureFormat.RGB24, false);
#endif
                }

                texture.LoadRawTextureData(pixels);
                texture.Apply();

                if (rawImage != null)
                {
                    rawImage.texture = texture;
                    rawImage.material.mainTexture = texture;
                }

                if (Mesh != null) Mesh.GetComponent<Renderer>().material.mainTexture = texture;
            }
            yield return null;
            LoadingTexture = false;
        }
    }

    private void UnregisterFormat()
    {
        Debug.Log("Unregistering camera pixel format " + mPixelFormat.ToString());
        CameraDevice.Instance.SetFrameFormat(mPixelFormat, false);
        mFormatRegistered = false;
    }

    private void RegisterFormat()
    {
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
        {
            Debug.Log("Successfully registered camera pixel format " + mPixelFormat.ToString());
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register camera pixel format " + mPixelFormat.ToString());
            mFormatRegistered = false;
        }
    }
}

A great solution for reading the pixels. Do you know how can I write the pixels back to the frame? — Derzu, Mar 27 '21 at 21:41
If I remember correctly, the webcam pixels are read only. btw, you can still change the shader of its background plane for effect or post processing. — thelghome, Mar 29 '21 at 04:09

score 0 · Answer 2 · answered Sep 06 '19 at 16:49

I was able to resolve this issue by working with the Vuforia ARCamera game object directly, rather than the BackgroundPlane Mesh Renderer. The ARCamera does not have a targetTexture set; as it outputs directly to the screen. However, I am able to set a temporary targetTexture to which to output a frame and then remove it (the targetTexture) immediately after processing, so that the AR mode may continue.

There is also a further solution. This is to use the TextureBufferCamera, which is created by Vuforia at runtime. This already outputs to a targetTexture. But, it is a fixed resolution and therefore the ARCamera is better for my specific requirement.

Capturing a snapshot of VideoBackground within Unity Vuforia plugin

2 Answers2