How to get preview buffer of MediaCapture - Universal app

Question

In Windows phone silverlight, I use PhotoCamera to get buffer frame when start preview video, in universal app I use MediaCapture instead, but I don't know how to get preview buffer.

Thanks

Answer 1

Since Windows Runtime doesn't have Silverlight's PhotoCaptureDevice class, the extremely useful GetPreviewBufferARGB() and GetPreviewBufferYCbCr() methods are not available.

The solution you're looking for is to use the MediaCapture.StartPreviewToCustomSinkAsync() method, but this requires C++ skills better than mine. No-one seems to have solved the problem and shared their code.

Now there's a really beautiful solution, using the Lumia Imaging SDK, that doesn't use the MediaCapture class, but will probably solve your problem even better.

Check out Microsoft's example on Github first. This works well but is quite complicated because it targets both Windows 8.1 and Windows Phone 8.1.

I've written some simpler code, just targeting Windows Phone, for my own understanding. It might help.

Start with a new C# Windows Phone 8.1 (Store) app with the Lumia Imaging SDK installed via NuGet PM. This example draws to an image element with x:Name="previewImage" in MainPage.xaml so make sure you add that. You'll also need to make the relevant imports to MainPage.xaml.cs which I think are.

using Lumia.Imaging;
using System.Threading.Tasks;
using Windows.UI.Xaml.Media.Imaging;
using Windows.UI.Core;
using System.ComponentModel;

Then you just add the following in the right place in MainPage.xaml.cs.

private CameraPreviewImageSource _cameraPreviewImageSource; // Using camera as our image source
private WriteableBitmap _writeableBitmap;
private FilterEffect _effect;
private WriteableBitmapRenderer _writeableBitmapRenderer; // renderer for our images
private bool _isRendering = false; // Used to prevent multiple renderers running at once

public MainPage()
{
    this.InitializeComponent();
    this.NavigationCacheMode = NavigationCacheMode.Required;
    startCameraPreview();
}

private async Task startCameraPreview()
{
    // Create a camera preview image source (from the Lumia Imaging SDK)
    _cameraPreviewImageSource = new CameraPreviewImageSource();
    await _cameraPreviewImageSource.InitializeAsync(string.Empty); // use the first available camera (ask  me if you want code to access other camera)
    var previewProperties = await _cameraPreviewImageSource.StartPreviewAsync();
    _cameraPreviewImageSource.PreviewFrameAvailable += drawPreview; // call the drawPreview method every time a new frame is available

    // Create a preview bitmap with the correct aspect ratio using the properties object returned when the preview started.
    var width = 640.0;
    var height = (width / previewProperties.Width) * previewProperties.Height;
    var bitmap = new WriteableBitmap((int)width, (int)height);
    _writeableBitmap = bitmap;

    // Create a BitmapRenderer to turn the preview Image Source into a bitmap we hold in the PreviewBitmap object
    _effect = new FilterEffect(_cameraPreviewImageSource);
    _effect.Filters = new IFilter[0]; // null filter for now
    _writeableBitmapRenderer = new WriteableBitmapRenderer(_effect, _writeableBitmap);
}

private async void drawPreview(IImageSize args)
{
    // Prevent multiple rendering attempts at once
    if (_isRendering == false)
    {
        _isRendering = true;
        await _writeableBitmapRenderer.RenderAsync(); // Render the image (with no filter)
        // Draw the image onto the previewImage XAML element
        await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(CoreDispatcherPriority.High,
            () =>
            {
                previewImage.Source = _writeableBitmap; // previewImage is an image element in MainPage.xaml
                _writeableBitmap.Invalidate(); // force the PreviewBitmap to redraw
            });
        _isRendering = false;
    }
}

You might be wondering... how do I grab the previewBuffer? You don't need to!

The _writeableBitmap object always holds the latest frame from the camera so you can do whatever you like with it.

Answer 2

My other answer is still valid for Universal Apps targeting Windows 8.1 -- but for those targeting Windows 10 with UWP apps there is a much simpler answer now.

Grabbing a preview frame in UWP is easy and well-documented -- from an existing MediaCapture object it can be done in three lines.

// Get information about the preview
var previewProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;

// Create a video frame in the desired format for the preview frame
VideoFrame videoFrame = new VideoFrame(BitmapPixelFormat.Bgra8, (int)previewProperties.Height, (int)previewProperties.Width);

// Grave a preview frame        
VideoFrame previewFrame = await _mediaCapture.GetPreviewFrameAsync(videoFrame);