Stenciling using OpenGl ES 2.0

Question

I am trying to figure out a way to cut out a certain region of a background texture such that a certain custom pattern is not rendered on the screen for that background. For example: enter image description here

This square can be any pattern. I am using Frame Buffer Object and Stencil Buffer to achieve this kind of effect. Here is the code:

fbo.begin();
//Disables ColorMask and DepthMask so that all the rendering is done on the Stencil Buffer
Gdx.gl20.glColorMask(false, false, false, false);
Gdx.gl20.glDepthMask(false);
Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);

stage.getSpriteBatch().begin();
rHeart.draw(stage.getSpriteBatch(), 1); //Draws the required pattern on the stencil buffer

//Enables the ColorMask and DepthMask to resume normal rendering
Gdx.gl20.glColorMask(true, true, true, true);
Gdx.gl20.glDepthMask(true);

Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

background.draw(stage.getSpriteBatch(), 1); //Draws the background such that the background is not rendered on the required pattern, leaving that area black.

stage.getSpriteBatch().end();
Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);
fbo.end();

However this is not working at all. How am I supposed to do this using Stencil Buffers? I am also facing some difficulty understanding glStencilFunc and glStencilOp. It would be very helpful if anyone can shed some light on these two.

UPDATE: I have also tried producing something of the same kind using glColorMask. Here is the code:

Gdx.gl20.glClearColor(0, 0, 0, 0);
stage.draw();
FrameBuffer.clearAllFrameBuffers(Gdx.app);
fbo1.begin();
Gdx.gl20.glClearColor(0, 0, 0, 0);
batch.begin();
rubber.draw(batch, 1);
Gdx.gl20.glColorMask(false, false, false, true);
coverHeart.draw(batch, 1);
Gdx.gl20.glColorMask(true, true, true, false);
batch.end();        
fbo1.end();

toDrawHeart = new Image(new TextureRegion(fbo1.getColorBufferTexture()));
batch.begin();
toDrawHeart.draw(batch, 1);
batch.end();

This code is producing this: enter image description here Instead of something like this: (Ignore the windows sizes and colour tones)

Note: I am using the libgdx library.

Can you elaborate on what 'not working at all' means in your case? — Tim, Jun 15 '12 at 18:49
@Tim It means that the background that is getting rendered doesn't contain any "pattern" that has been stencilled out. That means I am getting a completely pink screen rather than a pink with a black box in it (as shown above). — Rafay, Jun 15 '12 at 19:01
Also note that I am using the libgdx library. For the Desktop, I am using LwjglApplication, setting the stencil value to 8 for the LwjglApplicationConfiguration Object. — Rafay, Jun 15 '12 at 19:03
You're sure that stenciling works with the `SpriteBatch`? I'd a brief look at the [code](http://code.google.com/p/libgdx/source/browse/trunk/gdx/src/com/badlogic/gdx/graphics/g2d/SpriteBatch.java), and I'm not sure. — Stefan Hanke, Jun 16 '12 at 05:55
I am not sure about it. Well as an alternative, what should I try then? And why would you think that SpriteBatch might not work with Stencil Buffers? — Rafay, Jun 16 '12 at 06:13
These draw methods do not actually draw anything, but store the vertex coords, color and tex coords inside an array. On calling end(), renderMesh() is called. There are other renderMesh() calls that look plain weird. So I think even if the code changes OpenGL ES state, the only state that matters is the one active at the end() call. — Stefan Hanke, Jun 16 '12 at 19:42
Did you make sure that your GL context has been created with stencil bits in pixel format ? — rotoglup, Jun 16 '12 at 22:21
How is your `rHeart.draw` call drawing the pattern on the stencil buffer ? — rotoglup, Jun 16 '12 at 22:22
@rotoglup Well I am doing this for the desktop application: `public class Main { public static void main(String[] args) { JoglApplicationConfiguration cfg = new JoglApplicationConfiguration(); cfg.title = "FBOTest"; cfg.useGL20 = true; cfg.width = 720; cfg.height = 480; cfg.stencil = 8; new JoglApplication(new FBOTestGame(), cfg); } }` I think this is what you mean with the GL context. Here check this: http://libgdx.l33tlabs.org/docs/api/com/badlogic/gdx/scenes/scene2d/ui/Image.html Also here is the api implementation of the Image class: https://gist.github.com/2943895 — Rafay, Jun 17 '12 at 08:33
@StefanHanke So what should I do instead of calling these drawing methods inside the fbo? — Rafay, Jun 17 '12 at 08:35

Stefan Hanke · Accepted Answer · 2012-06-18T20:46:34.973

While drawing to a SpriteBatch, state changes are ignored, until end() is called. If you want to use stenciling with SpriteBatch, you'll need to break up the batch drawing. One thing, I've left out FBOs, but that shouldn't make a difference.

@Override
public void create() {      
    camera = new OrthographicCamera(1, 1);
    batch = new SpriteBatch();

    texture = new Texture(Gdx.files.internal("data/badlogic.jpg"));
    texture.setFilter(TextureFilter.Linear, TextureFilter.Linear);

    TextureRegion region = new TextureRegion(texture, 0, 0, 256, 256);

    sprite = new Sprite(region);
    sprite.setSize(1f, 1f);
    sprite.setPosition(-0.5f, -0.5f);

    spriteUpsideDown = new Sprite(new TextureRegion(texture, 1f, 1f, 0f, 0f));
    spriteUpsideDown.setSize(1f, 1f);
    spriteUpsideDown.setPosition(-0.5f, -0.5f);

    pattern = new Sprite(region);
    pattern.setSize(0.5f, 0.5f);
    pattern.setPosition(-0.25f, -0.25f);

    << Set Input Processor >>
}

The input processor allows to set two boolean flags breakBatch1 and breakBatch2 via keyboard (libgdx on desktop), which are used to break the SpriteBatch drawing.

@Override
public void render() {      
    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_STENCIL_BUFFER_BIT);
    batch.setProjectionMatrix(camera.combined);

    // setup drawing to stencil buffer
    Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
    Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff);
    Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    Gdx.gl20.glColorMask(false, false, false, false);

    // draw base pattern
    batch.begin();
    pattern.draw(batch);

    if(breakBatch1) { batch.end(); batch.begin(); }

    // fix stencil buffer, enable color buffer
    Gdx.gl20.glColorMask(true, true, true, true);
    Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

    // draw where pattern has NOT been drawn
    Gdx.gl20.glStencilFunc(GL20.GL_NOTEQUAL, 0x1, 0xff);
    sprite.draw(batch);

    if(breakBatch2) { batch.end(); batch.begin(); }

    // draw where pattern HAS been drawn.
    Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff);
    spriteUpsideDown.draw(batch);
    batch.end();
}

This did not work at all for me. It creates a rectangular mask, not caring about the sprite's alpha value at all. — DKIT, Apr 02 '14 at 09:22

score 2 · Answer 2 · answered Jun 15 '12 at 17:42

2

Gdx.gl20.glStencilFunc(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);

These are not the right arguments to glStencilFunc. I think you mean glStencilOp here.

You need to use glGetError in your code, it will alert you to these kinds of errors.

answered Jun 15 '12 at 17:42

Tim

35,413
11
95
121

@Spoilt: Do you use glGetError? – Tim Jun 15 '12 at 17:44
Hmm I'm not sure then. I'm not experienced with stencil buffers, though I have a pretty good idea how it works and what you have there looks sane to me. – Tim Jun 15 '12 at 18:49

rotoglup · Answer 3 · 2013-02-07T14:55:48.257

1

I believe your problem is that your initial GL_REPLACE stencil operation is applied to all the drawn pixels by your rHeart.draw regardless of the shape of any texture applied on the quad.

Thus, the stencil value is applied to every pixel of your quads which gives your problem.

If the texture applied on your quad has an alpha channel, as GL_ALPHA_TEST is not supported, you could setup your shader to discard the totally transparent pixels, preventting them from being drawn to the stencil buffer.

edited Feb 07 '13 at 14:55

answered Jun 17 '12 at 18:59

rotoglup

5,223
25
37

1

OpenGL ES 2.0 implementation doesn't allow GL_ALPHA_TEST and glAlphaFunc – Rafay Feb 07 '13 at 07:41

Stenciling using OpenGl ES 2.0

3 Answers3

Linked