OpenGL Stencil - Exclude transparent pixels

Question

I have this texture that I want to use as as mask using the Stencil buffer :

Then I want to draw an image IMG on the screen which should appear only where the above image is visible (where there's color), thus excluding transparent pixels above the top edge and below the gradient.

The problem is whenever I draw my image IMG on the screen it appears everywhere where the image has been drawn, whether the pixels are transparent or not.

So I thought about using the ALPHA_TEST but it's gone in the latest OpenGL versions, so I tried to discard fragment with v_color.a == 0 or v_color.r == 0 in my fragment shader but it does nothing...

Fragment shader :

#ifdef GL_ES
    #define LOWP lowp
    precision mediump float;
#else
    #define LOWP
#endif

uniform sampler2D u_texture;

varying LOWP vec4 v_color;
varying vec2 v_texCoords;

void main() {
    if (v_color.a == 0) {
        discard;
    }

    gl_FragColor = v_color * texture2D(u_texture, v_texCoords);
}

Vertex shader :

attribute vec4 a_color;
attribute vec4 a_position;
attribute vec2 a_texCoord0;

uniform mat4 u_projTrans;

varying vec4 v_color;
varying vec2 v_texCoords;

void main() {
    v_color = a_color;
    v_texCoords = a_texCoord0;
    gl_Position = u_projTrans * a_position;
}

Code :

@Override
public void draw(final DrawConfig drawConfig) {
    this.applyTransform(drawConfig.getBatch(), this.computeTransform());

    // Draw mask to stencil buffer

    drawConfig.end();
    final GL20 gl = Gdx.gl;

    gl.glEnable(GL20.GL_STENCIL_TEST);
    gl.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFF);
    gl.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    gl.glStencilMask(0xFF);
    gl.glColorMask(false, false, false, false);
    gl.glClearStencil(0);
    gl.glClear(GL20.GL_STENCIL_BUFFER_BIT);

    drawConfig.begin(Mode.BATCH);
    this.mBackground.setClippingEnabled(true); // The clipping area is a rectangle centered in the screen
    this.mBackground.draw(drawConfig);
    this.mBottomBorder.draw(drawConfig); // Border below the rectangle area
    this.mTopBorder.draw(drawConfig); // Border above the rectangle area
    drawConfig.end();

    gl.glColorMask(true, true, true, true);
    gl.glStencilFunc(GL20.GL_EQUAL, 1, 0xFF);
    gl.glStencilMask(0x00);

    // Draw elements

    this.mBackground.setClippingEnabled(false); // No clipping area, use stencil test
    this.mBackground.draw(drawConfig);
    this.mBottomBorder.draw(drawConfig);
    this.mTopBorder.draw(drawConfig);

    drawConfig.end();
    Gdx.gl.glDisable(GL20.GL_STENCIL_TEST);

    this.resetTransform(drawConfig.getBatch());
}

Result :

The red arrows show the parts that go outside of the borders and that I want to make disappear. The yellow square is the clipping area, for your convenience.

That image has partial transparency, and stencil tests are afaik all-or-nothing. In any case, post your code. — Colonel Thirty Two, Mar 27 '15 at 15:55
If you're just using the stencil buffer to clip another texture you can do the blending yourself in the fragment shader. That being said, discarding the pixel based on alpha should work. Maybe post the code that you tried to use? — Mokosha, Mar 27 '15 at 15:59
Given texture filtering, you may not want to do a binary ` == 0` test on the sampled alpha value. Consider making it something like `if (v_color.a <= 0.01)`. However, now that you have posted your actual code, it's clear that `v_color` has nothing to do with texture alpha - nevertheless that value is going to be interpolated and it may never interpolate to exactly **0**. — Andon M. Coleman, Mar 27 '15 at 16:49
Okay, well you're trying to duplicate the effect of `GL_ALPHA_TEST`, right? That test _was_ performed after fragment shader evaluation on the final color, not on the vertex color. So you neeed to test `gl_FragColor.a` rather than `v_color.a` if you want to implement this yourself. — Andon M. Coleman, Mar 27 '15 at 17:00
@AndonM.Coleman It works ! Except that some pixels within the border are transparent too, but I'll change the texture and add an almost-invisible red & green value so they won't pass the test. If you want you can post it as an anwser so I'll accept it. — Mickäel A., Mar 27 '15 at 17:32

score 1 · Accepted Answer · answered Mar 28 '15 at 03:04

The fixed-function alpha test (GL_ALPHA_TEST) you are trying to duplicate was a per-fragment test. It used to occur after the fragment shader finished, and you can implement it yourself using discard.

However, your current solution is using the interpolated per-vertex alpha value, and that has nothing to do with the texture you are trying to use as a mask. You need to test the alpha value assigned to gl_FragColor to do this properly:

void main() {
    gl_FragColor = v_color * texture2D(u_texture, v_texCoords);

    if (gl_FragColor.a == 0.0) {
        discard;
    }
}

That is the modern shader-based equivalent of:

glAlphaFunc (GL_NOTEQUAL, 0.0f); // Reject fragments with alpha == 0.0
glEnable    (GL_ALPHA_TEST);

OpenGL Stencil - Exclude transparent pixels

1 Answers1

Linked