YUV to RGB by Shader on iPhone

Question

Currently I am developing a video player using FFMPEG. I'm trying to convert YUV420P to RGB by Shader to reduce performance hit and I could see it works fine. The problem is caused when I try to change image size.

Case 1. YUV to RGB is perfect. but the image is not exactly fit to Texture Bounds. For example, if i play 640x360 video, right (640-512) part is cropped and bottom (512-360) is filled with green colored rectangle.

FRAME_X=512; //This is texture size
FRAME_Y=512;

    avpicture_fill((AVPicture *) f, [currentVideoBuffer.data mutableBytes],
               enc->pix_fmt,
               FRAME_X, FRAME_Y);

    av_picture_copy((AVPicture *) f, (AVPicture *) avFrame,
                enc->pix_fmt,
                enc->width, enc->height);

....

int yuvWidth= FRAME_X ;
int  yuvHeight= FRAME_Y;
glBindTexture ( GL_TEXTURE_2D, textureIdY );
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, 
yuvWidth, yuvHeight, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, y_channel);


glBindTexture ( GL_TEXTURE_2D, textureIdU );
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE,
yuvWidth/2, yuvHeight/2, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, u_channel);

glBindTexture ( GL_TEXTURE_2D, textureIdV );
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, 
yuvWidth/2, yuvHeight/2, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, v_channel);

Case 2. If i set actual image size to texture size, then image is exactly fit to texture but the color of image is a little bit strange. It has too much green color.

Does anybody give me some clues for this?? Thanks in advance.

Answer 1

First, you need to set proper wrap mode for non-power-of two textures.

glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

then, try to set filtering mode to GL_LINEAR

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);  
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);  

And shaders.

1. vertex shader

   attribute vec4 position;  
   attribute vec2 texcoord;  
   uniform mat4 modelViewProjectionMatrix;  
   varying vec2 v_texcoord;
   
   void main()  
   {  
     gl_Position = modelViewProjectionMatrix * position;  
     v_texcoord = texcoord.xy;  
   }  
   

2. set vertices

   vertices[0] = -1.0f;  // x0
   vertices[1] = -1.0f;  // y0
   vertices[2] =  1.0f;  // ..
   vertices[3] = -1.0f;
   vertices[4] = -1.0f;
   vertices[5] =  1.0f;
   vertices[6] =  1.0f;  // x3
   vertices[7] =  1.0f;  // y3
   

3. load a proper viewProj matrix, for 2d images (sprites), better to use ortho-projection.

   GLfloat modelviewProj[16];  
   mat4f_LoadOrtho(-1.0f, 1.0f, -1.0f, 1.0f, -1.0f, 1.0f, modelviewProj);  
   glUniformMatrix4fv(_uniformMatrix, 1, GL_FALSE, modelviewProj);  
   

4. next, make sure you using proper text coord, like this

   GLfloat texCoords[] = {  
     0.0f, 1.0f,  
     1.0f, 1.0f,  
     0.0f, 0.0f,  
     1.0f, 0.0f,  
   };  
   

5. at last, fragment shader like this:

   varying highp vec2 v_texcoord;  
   uniform sampler2D s_texture_y;  
   uniform sampler2D s_texture_u;  
   uniform sampler2D s_texture_v;
   
   void main()  
   {  
     highp float y = texture2D(s_texture_y, v_texcoord).r;  
     highp float u = texture2D(s_texture_u, v_texcoord).r - 0.5;  
     highp float v = texture2D(s_texture_v, v_texcoord).r - 0.5;  
     highp float r = y +             1.402 * v;  
     highp float g = y - 0.344 * u - 0.714 * v;  
     highp float b = y + 1.772 * u;       
     gl_FragColor = vec4(r,g,b,1.0);  
   }  
   

You may look at my code as an example of using FFmpeg and OpenGLES on iOS: https://github.com/kolyvan/kxmovie.