9

I have to convert and display YUV420P images to RGB colorspace using the AMD GPU on a Freescale iMX53 processor (OpenGL ES 2.0, EGL). Linux OS, no X11. To achieve this I should be able to create an appropriate image holding the YUV420P data: this could be either a YUV420P/YV12 image type or 3 simple 8-bit images, one for each component (Y, U, V).

glTexImage2D is excluded, because it's slow, the YUV420P frames are the results of a real time video decoding @25FPS and with glTexImage2D we can't keep the desired framerate.

There's an alternative: eglCreateImageKHR/glEGLImageTargetTexture2DOES. The only problem is that these can't handle any image format that would be suitable for YUV420/YV12 data.

EGLint attribs[] = {
  EGL_WIDTH, 800,
  EGL_HEIGHT, 480,
  EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_YV12_FSL,
  EGL_NONE
};

EGLint const req_attribs[] = {
  EGL_RED_SIZE, 5,
  EGL_GREEN_SIZE, 6,
  EGL_BLUE_SIZE, 5,
  EGL_ALPHA_SIZE, 0,
  EGL_SAMPLES, 0,
  EGL_COLOR_BUFFER_TYPE, EGL_RGB_BUFFER,
  EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
  EGL_NONE
};

...

display = eglGetDisplay(EGL_DEFAULT_DISPLAY);
eglInitialize(display, NULL, NULL);
eglBindAPI(EGL_OPENGL_ES_API);
eglChooseConfig(display, req_attribs, config, ARRAY_SIZE(config), &num_configs);
ctx = eglCreateContext(display, curr_config, NULL, NULL);
surface = eglCreateWindowSurface(display, curr_config, fb_handle, NULL);

...

EGLImageKHR yuv_img = eglCreateImageKHR(display, EGL_NO_CONTEXT, EGL_NEW_IMAGE_FSL, NULL, attribs); 
eglQueryImageFSL(display, yuv_img, EGL_CLIENTBUFFER_TYPE_FSL, (EGLint *)&ptr);
glGenTextures(1, &texture);
glBindTexture(GL_TEXTURE_2D, texture);
glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, yuv_img);

glEGLImageTargetTexture2DOES(...) fails. If I change the appropriate line in 'attribs' to this:

EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_RGB_565_FSL,

then the image can be assigned to an OpenGL ES texture, but it's not appropriate to hold either 8-bit data (Y/U/V) or a YUV420/YV12 data. Searching the net (including Freescale community forum) I've haven't found any solution to this.

How can I create an image which:

  • is fast to create;
  • eventually can be assigned to an already existing buffer (physical address or virtual address is given);
  • can be used in the fragment/vertex shader program to perform YUV --> RGB conversion;

Constraint is to avoid unneccessary memcpy(...)s due to performance reasons.

tselmeci
  • 125
  • 1
  • 1
  • 11

1 Answers1

8

I have implemented this on the i.MX53 for several YUV formats and it works really well. I have a published article about it, although it was generalized to cover more Android platforms:

http://software.intel.com/en-us/articles/using-opengl-es-to-accelerate-apps-with-legacy-2d-guis

I suspect your problem is that you are not binding to the correct texture target. It should be like this:

glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, hEglImage[iTextureIndex]);

glBindTexture(GL_TEXTURE_EXTERNAL_OES, hTexture[iIndex]);

And the eglImageAttributes should be one of these:

EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_YV12_FSL, EGL_NONE};
EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_NV21_FSL, EGL_NONE};
EGLint eglImageAttributes[] = {EGL_WIDTH, iTextureWidth, EGL_HEIGHT, iTextureHeight, EGL_IMAGE_FORMAT_FSL, EGL_FORMAT_YUV_UYVY_FSL, EGL_NONE};

hEglImage[iTextureIndex] = eglCreateImageKHR(eglDisplay, EGL_NO_CONTEXT, EGL_NEW_IMAGE_FSL, NULL, eglImageAttributes);

struct EGLImageInfoFSL EglImageInfo;
eglQueryImageFSL(eglDisplay, hEglImage[iTextureIndex], EGL_CLIENTBUFFER_TYPE_FSL, (EGLint *)&EglImageInfo);

Although this feature of the Freescale i.MX53 platform makes YUV to RGB color space conversion for video extremely fast, it does have a couple of limitations:

  1. It only supports those 3 YUV formats.
  2. eglCreateImageKHR() must allocate the buffers. There is no way to make it use existing buffers. Freescale confirmed that the NULL pointer can not be anything else, which technically violates the Khronos specification.

Freescale has resolved these problems on the i.MX6 platform, although the architecture is really different. Hope this helps.

ClayMontgomery
  • 2,786
  • 1
  • 15
  • 14
  • Thanks a lot for your help, that has proven to be a really precious bit of information. I'm rather new to OpenGL (ES) and got confused by the many new details I have to keep in mind to solve even such a simple issue. My YUV420 --> RGB converter now works, although not too efficiently, since uses 3 EGL_FORMAT_YUV_YV12_FSL images to hold Y, U and V components. I wasn't able to get the Y, U and V components out of a single EGL_FORMAT_YUV_YV12_FSL texture correctly in the fragment shader code, and this another approach just works... – tselmeci Feb 14 '14 at 10:38
  • By the way I'm rather sad that on iMX53 I can't pass the address of an existing buffer to eglCreateImageKHR(). Perhaps I'll approach this problem from reverse direction: I create many images and pass those images' physical addresses to VPU to put the decoded frame into. I'm not sure it'll work (e.g. DMA-memory is needed for VPU)... – tselmeci Feb 14 '14 at 10:45
  • That is what I did. Let the EGL allocate the buffers, then use those buffers to receive decoded frames from the VPU. I found that it eliminates the need to use the IPU and works even better. The CPU usage is very low. You can modify Freescale's gstreamer isink plugin or the mxc-vpu-test unit test. – ClayMontgomery Feb 14 '14 at 17:06
  • No success. VPU framebuffer addresses for Y, Cb and Cr are from eglQueryImageFSL. VPU decodes a frame, I do a memcpy from the virtual address provided by eglQueryImageFSL to video framebuffer and it just shows the same garbage over and over again. Phy addresses are starting from 0x00100000 (1MB), perhaps they're offsets _inside_ the GPU memory region? Now I simply pass these addresses to VPU... – tselmeci Feb 17 '14 at 10:58
  • Okay, I've figured it out! :) The 'gpu_nommu' kernel option must be used, so the physical addresses will be systemwide accessible (not only by the GPU using its MMU). – tselmeci Feb 17 '14 at 12:35
  • Good point. You also must have the latest EGL driver from Freescale. Mine is libEGL.so 270933 2011-11-16. You should not need any memcpy() at all. – ClayMontgomery Feb 17 '14 at 16:41