How to change orientation of camera preview callback buffer?

Question

This is a variation on a question often asked hereabouts but I don't see this exact situation, so I'll throw it out there.

I have an onPreviewFrame callback set up. This gets a byte[] with NV21 data in it. We h.264 encode it and send it out as a video stream. On the other side, we see the video skewed, either 90 or 270 degrees, depending on the phone.

So the question is, how to rotate the data, not just the preview image? Camera.Parameters.setRotation only affects taking the picture, not video. Camera.setDisplayOrientation specifically says it only affects the displaying preview, not the frame bytes:

This does not affect the order of byte array passed in onPreviewFrame(byte[], Camera), JPEG pictures, or recorded videos.

So is there a way, at any API level, to change the orientation of the byte array? Failing that, can you even rotate the NV21 (YVU) format that this come in, or do I need to RGB it first?

Maybe I'm missing something, but isn't it a relatively straight forward task of running the data through a matrix to rotate it whichever way you want? Presumably you know the width and height of your preview, so, you should be able to just copy the bytes over into a new byte array that has them sorted in the desired orientation, no? I haven't done this, so, perhaps I'm woefully off base. — Yevgeny Simkin, Jan 28 '12 at 11:34
No, that would do it, but to do this for every frame, at even say 5ps, seems like a lot of processing. But I think it may be the only way. — Jonathan, Feb 24 '12 at 19:42

score 1 · Accepted Answer · answered Jun 04 '14 at 17:50

1

Turns out you do need to rotate each frame yourself before sending it off. We ended up using libyuv, which has a very convenient function that both rotates and converts it - libyuv::ConvertToI420

https://code.google.com/p/libyuv/

answered Jun 04 '14 at 17:50

Jonathan

2,532
3
16
16

score 0 · Answer 2 · answered Mar 24 '12 at 21:07

I think that you would need to rotate the picture yourself. I did it once using the NDK and the leptonica library. A look at my code should get you started. Performance was okayish on a Samsung Galaxy S2 (i think i got around 15 frames or so). Since i was pushing the result into an openGL texture i had to also swizzle the color bytes around.. You could speed it up by rotating the image directly in the loop which decodes the yuv data..

mPix32 and mPix8 were previously allocated to hold the converted data.You would need to replace with your own image data structure of course..

jint Java_de_renard_ImageFilter_nativeProcessImage(JNIEnv *env, jobject javathis, jbyteArray frame) {
    ....
    jbyte *data_buffer = env->GetByteArrayElements(frame, NULL);
    l_uint8 *byte_buffer = (l_uint8 *) data_buffer;
    yuvToPixFast(byte_buffer, mPix32, mPix8);
    env->ReleaseByteArrayElements(frame, data_buffer, JNI_ABORT);
    ....
}

static inline void yuvToPixFast(unsigned char* pY, Pix* pix32, Pix* pix8) {
    int i, j;
    int nR, nG, nB;
    int nY, nU, nV;
    l_uint32* data = pixGetData(pix32);
    l_uint32* data8 = pixGetData(pix8);
    l_int32 height = pixGetHeight(pix32);
    l_int32 width = pixGetWidth(pix32);
    l_int32 wpl = pixGetWpl(pix32);
    l_int32 wpl8 = pixGetWpl(pix8);
    l_uint8 **lineptrs = pixSetupByteProcessing(pix8, NULL, NULL);
    l_uint8* line8;

    //memcpy(data8,pY,height*width);

    unsigned char* pUV = pY + width * height;

    for (i = 0; i < height; i++) {
        nU = 0;
        nV = 0;
        unsigned char* uvp = pUV + (i >> 1) * width;
        line8 = lineptrs[i];
        memcpy(line8, pY, wpl8 * 4);

        for (j = 0; j < width; j++) {

            if ((j & 1) == 0) {
                nV = (0xff & *uvp++) - 128;
                nU = (0xff & *uvp++) - 128;
            }
            // Yuv Convert
            nY = *(pY++);
            //*line8++ = (l_uint8) nY;
            nY -= -16;

            if (nY < 0) {
                nY = 0;
            }
            int y1192 = nY * 1192;

            /*double saturation to increase cartoon effect*/
            //nU<<=1;
            //nV<<=1;

            nB = y1192 + 2066 * nU;
            nG = y1192 - 833 * nV - 400 * nU;
            nR = y1192 + 1634 * nV;

            if (nR < 0) {
                nR = 0;
            } else if (nR > 262143) {
                nR = 262143;
            }
            if (nG < 0) {
                nG = 0;
            } else if (nG > 262143) {
                nG = 262143;
            }
            if (nB < 0) {
                nB = 0;
            } else if (nB > 262143) {
                nB = 262143;
            }
            //RGBA
            //ABGR
            *data++ = ((nR << 14) & 0xff000000) | ((nG << 6) & 0xff0000) | ((nB >> 2) & 0xff00) | (0xff);
            //*data++ = (0x00 << 24) | (0xff<<16) | (0x00<<8) | ( 0xff) ;
            //*data++ = (0xff << 24) | ((nB << 6) & 0xff0000) | ((nG >> 2) & 0xff00) | ((nR >> 10) & 0xff);
        }
    }
    pixCleanupByteProcessing(pix8, lineptrs);

}

How to change orientation of camera preview callback buffer?

2 Answers2

Linked