Yuv (NV21) image converting to bitmap

Question

I am trying to capture images from camera preview and do some drawing on it. The problem is, I have only about 3-4 fps of drawing, and half of the frame processing time is receiving and decoding NV21 image from camera preview and converting to bitmap. I have a code to do this task, which I found on another stack question. It does not seem to be fast, but I do not know how to optimize it. It takes about 100-150 ms on Samsung Note 3, image size 1920x1080. How can I make it work faster?

Code :

public Bitmap curFrameImage(byte[] data, Camera camera)
{
    Camera.Parameters parameters = camera.getParameters();
    int imageFormat = parameters.getPreviewFormat();

    if (imageFormat == ImageFormat.NV21)
    {
        YuvImage img = new YuvImage(data, ImageFormat.NV21, prevSizeW, prevSizeH, null);
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        img.compressToJpeg(new android.graphics.Rect(0, 0, img.getWidth(), img.getHeight()), 50, out);
        byte[] imageBytes = out.toByteArray();
        return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.length);
    }
    else
    {
        Log.i(TAG, "Preview image not NV21");
        return null;
    }
}

The final format of image has to be bitmap, so I could then do processing on it. I've tried to set Camera.Parameters.setPreviewFormat to RGB_565, but could not assign camera params to camera, I've read also that NV21 is the only available format. I am not sure about that, whether it is possible to find solution in these format changes.

Thank you in advance.

Answer 1

Thank you, Alex Cohn, for helping me to do make this conversion faster. I implemented your suggested methods (RenderScript intrinsics). This code, made with RenderScript intrinsics, converts YUV image to bitmap about ~5 times faster. Previous code took 100-150 ms. on Samsung Note 3, this takes 15-30 or so. If someone needs to do the same task, here is the code:

These will be used:

private RenderScript rs;
private ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic;
private Type.Builder yuvType, rgbaType;
private Allocation in, out;

In on create function I initialize..:

rs = RenderScript.create(this);
yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

And the whole onPreviewFrame looks like this (here I receive and convert the image):

if (yuvType == null)
{
    yuvType = new Type.Builder(rs, Element.U8(rs)).setX(dataLength);
    in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);

    rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(prevSizeW).setY(prevSizeH);
    out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);
}

in.copyFrom(data);

yuvToRgbIntrinsic.setInput(in);
yuvToRgbIntrinsic.forEach(out);

Bitmap bmpout = Bitmap.createBitmap(prevSizeW, prevSizeH, Bitmap.Config.ARGB_8888);
out.copyTo(bmpout);

Answer 2

You can get even more speed (using JellyBean 4.3, API18 or higher): Camera preview mode must be NV21 !

On "onPreviewFrame()" do only:

aIn.copyFrom(data);

yuvToRgbIntrinsic.forEach(aOut);

aOut.copyTo(bmpout);  // and of course, show the bitmap or whatever

Do not create any objects here.

All other stuff (creating rs, yuvToRgbIntrinsic, allocations, bitmap) do in the onCreate() method before starting the camera preview.

rs = RenderScript.create(this);
yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs));

// you don´t need Type.Builder objects , with cameraPreviewWidth and cameraPreviewHeight do:
int yuvDatalength = cameraPreviewWidth*cameraPreviewHeight*3/2;  // this is 12 bit per pixel  
aIn = Allocation.createSized(rs, Element.U8(rs), yuvDatalength);

// of course you will need the Bitmap
bmpout = Bitmap.createBitmap(cameraPreviewWidth, cameraPreviewHeight, Bitmap.Config.ARGB_8888);

// create output allocation from bitmap
aOut = Allocation.createFromBitmap(rs,bmpout);  // this simple !

// set the script´s in-allocation, this has to be done only once
yuvToRgbIntrinsic.setInput(aIn);

On Nexus 7 (2013, JellyBean 4.3) a full HD (1920x1080) camera preview conversion takes about 0.007 s (YES, 7 ms).

Answer 3

OpenCV-JNI to construct Mat from Nv21 data for 4160x3120 sized image seems 2x faster (38msec) compared to renderscript (68msec-excluding initialization time). If we need to resize down constructed bitmap, OpenCV-JNI seems better approach as we would be using full size only for Y data. CbCr data would be resized to downsized Your data at the time of OpenCV mat construction only.

A still better way is passing NV21 byte array, and int pixel array to Jni. Array copy may not be needed at Jni side. Then, use open source libyuv library (https://chromium.googlesource.com/libyuv/libyuv/) to convert NV21 to ARGB. In Java, we use passed pixel array to construct bitmap. In Jni, conversion from NV21 to ARGB takes only ~4ms for 4160x3120 sized byte array on arm64 platform.