YUV (NV21) to BGR conversion on mobile devices (Native Code)

Question

I'm developing a mobile application that runs on Android and IOS. It's capable of real-time-processing of a video stream. On Android I get the Preview-Videostream of the camera via android.hardware.Camera.PreviewCallback.onPreviewFrame. I decided to use the NV21-Format, since it should be supported by all Android-devices, whereas RGB isn't (or just RGB565).

For my algorithms, which mostly are for pattern recognition, I need grayscale images as well as color information. Grayscale is not a problem, but the color conversion from NV21 to BGR takes way too long.

As described, I use the following method to capture the images;

In the App, I override the onPreviewFrame-Handler of the Camera. This is done in CameraPreviewFrameHandler.java:

 @Override
      public void onPreviewFrame(byte[] data, Camera camera) {
      {
          try {
              AvCore.getInstance().onFrame(data, _prevWidth, _prevHeight, AvStreamEncoding.NV21);
          } catch (NativeException e) 
          {
              e.printStackTrace();
          } 
      }

The onFrame-Function then calls a native function which fetches data from the Java-Objects as local references. This is then converted to an unsigned char* bytestream and calls the following c++ function, which uses OpenCV to convert from NV21 to BGR:

void CoreManager::api_onFrame(unsigned char* rImageData, avStreamEncoding_t vImageFormat, int vWidth, int vHeight)
    {
    // rImageData is a local JNI-reference to the java-byte-array "data" from onPreviewFrame
    Mat bgrMat; // Holds the converted image
    Mat origImg;    // Holds the original image (OpenCV-Wrapping around rImageData)

    double ts; // for profiling
    switch(vImageFormat)
    {
           // other formats
        case NV21:
            origImg = Mat(vHeight + vHeight/2, vWidth, CV_8UC1, rImageData); // fast, only creates header around rImageData
            bgrMat = Mat(vHeight, vWidth, CV_8UC3); // Prepare Mat for target image
            ts = avUtils::gettime();                // PROFILING START
            cvtColor(origImg, bgrMat, CV_YUV2BGRA_NV21);
            _onFrameBGRConversion.push_back(avUtils::gettime()-ts); // PROFILING END
            break;
    }

    [...APPLICATION LOGIC...]
}

As one might conclude from comments in the code, I profiled the conversion already and it turned out that it takes ~30ms on my Nexus 4, which is unacceptable long for such a "trivial" pre-processing step. (My profiling methods are double-checked and working properly for real-time measurement)

Now I'm trying desperately to find a faster implementation of this color conversion from NV21 to BGR. This is what I've already done;

Adopted the code "convertYUV420_NV21toRGB8888" to C++ provided in this topic (multiple of the conversion-time)
Modified the code from 1 to use only integer operations (doubled conversion-time of openCV-Solution)
Browsed through a couple other implementations, all with similar conversion-times
Checked OpenCV-Implementation, they use a lot of bit-shifting to get performance. Guess I'm not able to do better on my own

Do you have suggestions / know good implementations or even have a completely different way to work around this Problem? I somehow need to capture RGB/BGR-Frames from the Android-Camera and it should work on as many Android-devices as possible.

Thanks for your replies!

I am not entirely sure if I agree that software color conversion is a trivial pre-processing step, might be mathematically trivial but in terms of computation... you have to go through all the pixels. — Rui Marques, Sep 13 '13 at 09:38

score 1 · Answer 1 · answered Sep 12 '13 at 09:42

1

Did you try libyuv? I used it in the past and if you compile it with NEON support, it uses an asm code optimized for ARM processors, you can start from there to further optimize for your special situation.

answered Sep 12 '13 at 09:42

Dragos Iordache

139
3

Thanks for the hint, I haven't tried it yet and I'm not sure if I should; We try to write code that runs similar on every platform without adaption, thus we favor algorithm-optimization more. This may consider CPU-operation-optimization, for instance stuff like lookup-tables to minimize pipeline-stalls suits well because most architectures have similar jump-prediction-algorithms respectively jump-prediction is not necessary then. But in case we don't find a cross-platform way of optimizing this, I'll happily come back to your suggestion, thank you! – Thomas Bergmueller Sep 12 '13 at 10:29

YUV (NV21) to BGR conversion on mobile devices (Native Code)

1 Answers1