1

I have a Unity3D application that plays videos on the UI. I find myself in need to find a better way to convert a YUV video buffer to a RGB buffer. My situation is this:

  • Unity3D with a UI image that renders a video
  • Gstreamer external process which actually plays the video
  • A native plugin, called from Unity3D to convert the video YUV buffer to a RGBA one

My C++/Native plugin portion of code related to the YUV->RGBA conversion:

unsigned char * rgba = (unsigned char*) obj->g_RgbaBuff;
unsigned char * yuv =  (unsigned char*) obj->g_Buffer;

while ( ta < obj->g_BufferLength) 
{
    int ty = (int)yuv[i];
    int tu = (int)yuv[i + 1];
    int tY2 = (int)yuv[i + 2];
    int tv = (int)yuv[i + 3];

    int tp1 = (int)(1.164f * (ty - 16));
    int tr = Clamp((int)tp1 + 1.596f * (tv - 128));
    int tg = Clamp((int)tp1 - 0.813f * (tv - 128) - 0.391f * (tu - 128));
    int tb = Clamp((int)tp1 + 2.018f * (tu - 128));

    rgba[ta] = tb;
    rgba[ta + 1] = tg;
    rgba[ta + 2] = tr;
    ta += 4;



    int tp2 = (int)(1.164f * (tY2 - 16));
    int tr2 = Clamp((int)tp2 + 1.596f * (tv - 128));
    int tg2 = Clamp((int)tp2 - 0.813f * (tv - 128) - 0.391f * (tu - 128));
    int tb2 = Clamp((int)tp2 + 2.018f * (tu - 128));

    rgba[ta] = tb2;
    rgba[ta + 1] = tg2;
    rgba[ta + 2] = tr2;
    ta += 4;
}

This code gets called by Unity3D in a while loop to continuously update the output of my image, which is correctly showing the video. Thing is, it's really slow. When I'm opening more than one video, my FPSs drop from 60 to way below 30 with just three 720p videos. Is there a way to do this on the GPU? Or a smarter way to do it. Should I approach it in a different way?

To render the buffer to a texture I'm using this code in my native code, there's the rendering being done every frame by using Unity GL.IssuePluginEvent()

static void ModifyTexturePixels(void* textureHandle, int w, int h,  void* rgbaBuff)
{   
    int textureRowPitch;

    glBindTexture(GL_TEXTURE_2D, (GLuint)(size_t)textureHandle);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w, h, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, rgbaBuff);
}
AzeExMachina
  • 104
  • 14
  • 1
    I think you mean `YUV`? – apple apple Oct 16 '19 at 14:59
  • Yes, sorry, it was a typo, edited, thank you. – AzeExMachina Oct 16 '19 at 15:01
  • And yes, this can be done in shader pretty simple. (if all you want is render it) But have you check if the convert part really the bottleneck? – apple apple Oct 16 '19 at 15:02
  • Yes, I only need to render the video, I don't need to edit it or do fancy stuff. How can I do this? I've tried to use a shader like this https://stackoverflow.com/questions/54299351/yuv-shader-for-unity but I'm not seeing the texture. And also, that shader is not implementing UI masking, which I need. – AzeExMachina Oct 16 '19 at 15:05
  • IIRC, you needs to use that shader on a quad, not UI image. Also you can try about RenderTexture and bind it to UI image. (im not sure since I don't touch Unity for a while) – apple apple Oct 16 '19 at 15:20
  • just to add the reason, I don't know if UI image has proper UV coord. Maybe you can also try change the renderer on UI element. – apple apple Oct 16 '19 at 15:34

0 Answers0