Fetch raw data from an Image acquired from an AImageReader

Question

I have an ImageReader, whose surface is attached to a MediaCodec decoder to render into.

        AMediaCodec *videoDecoder = nullptr;
        ANativeWindow* surface = nullptr;

        AImageReader* imageReader = nullptr;
        AImageReader_ImageListener* imageListener = nullptr;

        if ((videoDecoder = AMediaCodec_createDecoderByType(mime)))
        {
            if (AImageReader_new(mWidth, mHeight, AIMAGE_FORMAT_YUV_420_888, 2, &imageReader) == AMEDIA_OK)
            {
                if (AImageReader_getWindow(imageReader, &surface) == AMEDIA_OK)
                {
                    if (AMediaCodec_configure(videoDecoder, mediaFormat, surface, NULL, 0) == AMEDIA_OK)
                    {
                        int32_t outputFormat{};
                        AMediaFormat_getInt32(AMediaCodec_getOutputFormat(videoDecoder), AMEDIAFORMAT_KEY_COLOR_FORMAT, &outputFormat);

                        imageListener = new AImageReader_ImageListener();
                        imageListener->onImageAvailable = &onImageAvailableCallback;
                        AImageReader_setImageListener(imageReader, imageListener);

                        if (AMediaCodec_start(videoDecoder) == AMEDIA_OK)
                        {
                            configCompleted = true;
                        }
                        else
                        {
                            TRACE("ImporterMP4Android", 0, "Failed to Start Video Decoder");
                        }
                    }
                    else
                    {
                        TRACE("ImporterMP4Android", 0, "Failed to Configure Video Decoder");
                    }
                }
                else
                {
                    TRACE("ImporterMP4Android", 0, "Failed to Fetch Surface owned by the ImageReader");
                }
            }
            else
            {
                TRACE("ImporterMP4Android", 0, "Failed to Create ImageReader");
            }
        }
        else
        {
            TRACE("ImporterMP4Android", 0, "Failed to Create Decoder");
        }

The onImageAvailableCallback looks like this atm:

void onImageAvailableCallback(void *context, AImageReader *reader)
{
    int32_t format;
    media_status_t  status = AImageReader_getFormat (reader, &format);

    AImage *image;
    status = AImageReader_acquireLatestImage(reader, &image);


    status = AImage_getFormat(image, &format);

    // TODO: copy *raw data somewhere for downstream processing

    AImage_delete(image);
}

As indicated in TODO comment, I want to copy out the raw data of the Image acquired from ImageReader for further processing. The interface provided by the Image class allows me to query the number of planes and fetch individual planar data, but I am interested in grabbing the entire frame at once. Any suggestions on how I might accomplish this?

In a nutshell, I am using MediaCodec video decoder to render into a Surface owned by an ImageReader, and ultimately want to grab the decoded video frames from the ImageReader in their entirety in YUV420NV21 format for further processing.

score 2 · Answer 1 · answered Dec 12 '17 at 07:43

If you want all three YUV planes, you need to copy them one by one into whatever destination buffer you want them in. You can't expect them to be contiguous, but all three planes can be placed arbitrarily far apart in memory. Something like this (untested) is pretty much what you'd need:

uint8_t *buf = new uint8_t[width*height + 2*(width+1)/2*(height+1)/2];
int32_t yPixelStride, yRowStride;
uint8_t *yPtr;
int yLength;
AImage_getPlanePixelStride(image, 0, &yPixelStride);
AImage_getPlaneRowStride(image, 0, &yRowStride);
AImage_getPlaneData(image, 0, &yPtr, &yLength);
if (yPixelStride == 1) {
    // All pixels in a row are contiguous; copy one line at a time.
    for (int y = 0; y < height; y++)
        memcpy(buf + y*width, yPtr + y*yRowStride, width);
} else {
    // Highly improbable, but not disallowed by the API. In this case
    // individual pixels aren't stored consecutively but sparsely with
    // other data inbetween each pixel.
    for (int y = 0; y < height; y++)
        for (int x = 0; x < width; x++)
            buf[y*width+x] = yPtr[y*yRowStride + x*yPixelStride];
}
int32_t cbPixelStride, crPixelStride, cbRowStride, crRowStride;
uint8_t *cbPtr, *crPtr;
int cbLength, crLength;
AImage_getPlanePixelStride(image, 1, &cbPixelStride);
AImage_getPlaneRowStride(image, 1, &cbRowStride);
AImage_getPlaneData(image, 1, &cbPtr, &cbLength);
AImage_getPlanePixelStride(image, 2, &crPixelStride);
AImage_getPlaneRowStride(image, 2, &crRowStride);
AImage_getPlaneData(image, 2, &crPtr, &crLength);
uint8_t *chromaBuf = &buf[width*height];
int chromaBufStride = 2*((width + 1)/2);
if (cbPixelStride == 2 && crPixelStride == 2 &&
    cbRowStride == crRowStride && crPtr == cbPtr + 1) {
    // The actual cb/cr planes happened to be laid out in
    // exact NV21 form in memory; copy them as is
    for (int y = 0; y < (height + 1)/2; y++)
        memcpy(chromabuf + y*chromaBufStride, crPtr + y*crRowStride, chromaBufStride);
} else if (cbPixelStride == 2 && crPixelStride == 2 &&
           cbRowStride == crRowStride && crPtr == cbPtr + 1) {
    // The cb/cr planes happened to be laid out in exact NV12 form
    // in memory; if the destination API can use NV12 in addition to
    // NV21 do something similar as above, but using cbPtr instead of crPtr.
    // If not, remove this clause and use the generic code below.
} else {
    if (cbPixelStride == 1 && crPixelStride == 1) {
        // Continuous cb/cr planes; the input data was I420/YV12 or similar;
        // copy it into NV21 form
        for (int y = 0; y < (height + 1)/2; y++) {
            for (int x = 0; x < (width + 1)/2; x++) {
                chromaBuf[y*chromaBufStride + 2*x + 0] = crPtr[y*crRowStride + x];
                chromaBuf[y*chromaBufStride + 2*x + 1] = cbPtr[y*cbRowStride + x];
            }
        }
    } else {
        // Generic data copying into NV21
        for (int y = 0; y < (height + 1)/2; y++) {
            for (int x = 0; x < (width + 1)/2; x++) {
                chromaBuf[y*chromaBufStride + 2*x + 0] = crPtr[y*crRowStride + x*crPixelStride];
                chromaBuf[y*chromaBufStride + 2*x + 1] = cbPtr[y*cbRowStride + x*cbPixelStride];
            }
        }
    }
}

However, many APIs can process data in the form of three pointers to the start of each plane, plus the row stride of each. Then you can get away with much less copying. For the chroma, you still probably want to try to identify whether it is I420/NV12/NV21 and pass it as-is to the destination API. If you can't match it to a specific pixel format layout that the destination API supports, you need to unpack it into generically into a local buffer with a known supported layout.

Thanks @mstorsjo. Ultimately, I followed the buffer approach (as against the ImageReader approach), with the MediaCodec decoder (configured with YUV420Planar format) giving me contiguous YUV data (width * height bytes for Y followed by width * height / 4 bytes for U and V each). This allowed copying out the YUV data in a custom data structure for further processing. — Mohit Bhola, Apr 03 '18 at 07:50

Fetch raw data from an Image acquired from an AImageReader

1 Answers1