OpenGL: Chromakey With White

Question

I am following this guide by Erik Buck to get a green screen effect for my video processing. This works absolutely marvelous but it doesn't exactly meet my requirement. I am tasked with using OpenGL in my iOS project to 'cut out' the white using this same method.

Below is the OpenGL code from Erik Buck's project to find the green texels for each texture and assign it's opacity to zero:

varying highp vec2 vCoordinate;
uniform sampler2D uVideoframe;


void main()
{
   // Look up the color of the texel corresponding to the fragment being
   // generated while rendering a triangle
   lowp vec4 tempColor = texture2D(uVideoframe, vCoordinate);

   // Calculate the average intensity of the texel's red and blue components
   lowp float rbAverage = tempColor.r * 0.5 + tempColor.b * 0.5;

   // Calculate the difference between the green element intensity and the
   // average of red and blue intensities
   lowp float gDelta = tempColor.g - rbAverage;

   // If the green intensity is greater than the average of red and blue
   // intensities, calculate a transparency value in the range 0.0 to 1.0
   // based on how much more intense the green element is
   tempColor.a = 1.0 - smoothstep(0.0, 0.25, gDelta);

   // Use the cube of the transparency value. That way, a fragment that
   // is partially translucent becomes even more translucent. This sharpens
   // the final result by avoiding almost but not quite opaque fragments that
   // tend to form halos at color boundaries.
   tempColor.a = tempColor.a * tempColor.a * tempColor.a;

      gl_FragColor = tempColor;
}

Through trial and error I was able to manipulate this code to be able to make different colors transparent, but white has been proven a challenge.

How do I make the white in the background to be transparent?

Also, can you provide a sample of the kind of image you’re trying to do this with? — Noah Witherspoon, Sep 25 '17 at 18:08
I added a picture of the white I am trying to cut out. Basically I am trying to create a green screen effect, but instead it needs to be a "white screen". — Brandon A, Sep 25 '17 at 18:17
@MichaelIvanov I think alpha blending is what I am doing. I am an iOS mobile engineer and not that savvy on OpenGL. — Brandon A, Sep 25 '17 at 18:20
@Rabbid76 the white background is the sheet of paper I am holding infront of the camera. I should be able to put my hands over the white sheet of paper and it cut out the white, leaving only my hands. — Brandon A, Sep 25 '17 at 18:21

MuertoExcobito · Answer 1 · 2017-09-27T02:19:06.193

1

You will need to modify the algorithm in the shader to change the color. The value of gDelta is determining factor in the opacity of the pixel output. In your posted code, once the value of the green component is higher than the average of the red and blue, the pixel starts to become transparent.

You essentially just need to decide which formula you want to use to detect 'white'. One very simple (and similar) way would be calculate the average of all components, and define some cutoff. For example:

//...
float cutoff = 0.8f;
lowp float gDelta = cutoff - dot(tempColor, vec4(0.33, 0.33, 0.33, 0.0));
//...

You will need to modify this slightly to come up with a color range that suits your needs.

Also, this process is formally call chroma-keying. Although you can use any color, generally colors that will not match with the "cutout" object. Bright green is a frequent choice, because it rarely matches with skin tones or clothing. For this reason, white might be a really bad choice.

edited Sep 27 '17 at 02:19

answered Sep 25 '17 at 18:58

MuertoExcobito

9,741
2
37
78

Thank you for your answer. I will try your suggestion. And I have already voiced my opinion for using white because it could affect the skin tone as well. But I am still having to make it work with that color. – Brandon A Sep 25 '17 at 20:47
My compiler says there is a syntax error. It doesn't say which line but I think it doesn't know what float3() is. – Brandon A Sep 25 '17 at 21:02
Then when I try and hardcore `0.33` I get this error: `No matching function for call to dot(vec4, float)` – Brandon A Sep 25 '17 at 21:08

score 1 · Accepted Answer · answered Jun 15 '18 at 19:46

White is actually quite simple:

void main()
{
    lowp vec4 tempColor = texture2D(uVideoframe, vCoordinate);
    lowp float rgbAverage = tempColor.r + tempColor.b + tempColor.g;
    if (rgbAverage > 1.99) {
        tempColor.a = 0.0;
    }
    else {
        tempColor.a = 1.0;
    }
    gl_FragColor = tempColor;
}

I found that setting the threshold to 1.99 adequately removes a white background without affecting the whiteness of my hand.

OpenGL: Chromakey With White

2 Answers2