1

I'm looking for a solution to implement alpha masking with stencil buffer in libgdx with open gles 2.0.

I have managed to implement simple alpha masking with stencil buffer and shaders, where if alpha channel of fragment is greater then some specified value it gets discarted. That works fine.

The problem is when I want to use some gradient image mask, or fethered png mask, I don't get what I wanned (I get "filled" rectangle mask with no alpha channel), instead I want smooth fade out mask.

I know that the problem is that in stencil buffer there are only 0s and 1s, but I want to write to stencil some other values, that represent actual alpha value of fragment that passed in fragment shader, and to use that value from stencil to somehow do some blending. I hope that I've explained what I want to get, actually if it's possible.

I've recently started playing with OpenGL ES, so I still have some misunderstandings.

My questions is: How to setup and stencil buffer to store values other then 0s and 1s, and how to use that values later for alpha masking?

Tnx in advance.

This is currently my stencil setup:

    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_STENCIL_BUFFER_BIT |   GL20.GL_DEPTH_BUFFER_BIT);

    // setup drawing to stencil buffer
    Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
    Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff);
    Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    Gdx.gl20.glColorMask(false, false, false, false);
    Gdx.gl20.glDepthMask(false);

    spriteBatch.setShader(shaderStencilMask);
    spriteBatch.begin();

    // push to the batch
    spriteBatch.draw(Assets.instance.actor1, Gdx.graphics.getWidth() / 2, Gdx.graphics.getHeight() / 2, Assets.instance.actor1.getRegionWidth(), Assets.instance.actor1.getRegionHeight());

    spriteBatch.end();

    // fix stencil buffer, enable color buffer
    Gdx.gl20.glColorMask(true, true, true, true);
    Gdx.gl20.glDepthMask(true);
    Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

    // draw where pattern has NOT been drawn
    Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff);

    decalBatch.add(decal);
    decalBatch.flush();

    Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);

    decalBatch.add(decal2);
    decalBatch.flush();
Veljko
  • 1,893
  • 6
  • 28
  • 58
  • Stenciling doesn't allow partial alpha values. To get in-between values I think you'll have to draw your stuff into a FrameBuffer and then draw to the screen with multi-texturing, using the FrameBuffer's color texture and a texture representing alpha. But there might be a simpler solution depending on your specific need. – Tenfour04 Jul 18 '14 at 15:00
  • Im trying to mask 3d planes, for example one plane represents mask, and other plane is behind the mask plane. I want behind plane to get transparent where mask's texture is transparent. Im currently using Decals in Libgdx. Is that even possible without fps drop? – Veljko Jul 18 '14 at 15:21
  • By planes, you mean rectangles in 3D space? If the two rectangles are exactly the same in size, orientation, and position then you could just do it with multi-texturing, although that would require a bit of a work-around with your decals. – Tenfour04 Jul 18 '14 at 15:29
  • Yes, planes are basically rectangles in 3d space. Mask should be static, and plane behind the mask can move around. And overlap parts should be masked. – Veljko Jul 18 '14 at 15:33
  • I can only think of ways that involve FrameBuffers. Whether it affects FPS depends on what your bottleneck is. – Tenfour04 Jul 18 '14 at 16:46
  • I think that real time frame buffer rendering and updating texture on 3d plane may be the bottle neck. What do you think? – Veljko Jul 18 '14 at 16:49
  • No way to predict without seeing your whole project. For example, if you have 10000 decals on screen, then the bottleneck would more likely be the CPU load of transferring the vertices, or overdraw of those blended decals, not drawing the FBO. – Tenfour04 Jul 18 '14 at 17:01
  • Ok. I will try that way...and see what happens. Tnx for advice. – Veljko Jul 18 '14 at 17:03

1 Answers1

2

The only ways I can think of doing this are with a FrameBuffer.

Option 1

Draw your scene's background (the stuff that will not be masked) to a FrameBuffer. Then draw your entire scene without masks to the screen. Then draw your mask decals to the screen using the FrameBuffer's color attachment. Downside to this method is that in OpenGL ES 2.0 on Android, a FrameBuffer can have RGBA4444, not RGBA8888, so there will be visible seams along the edges of the masks where the color bit depth changes.

Option 2

Draw you mask decals as B&W opaque to your FrameBuffer. Then draw your background to the screen. When you draw anything that can be masked, draw it with multi-texturing, multiplying by the FrameBuffer's color texture. Potential downside is that absolutely anything that can be masked must be drawn multi-textured with a custom shader. But if you're just using decals, then this isn't really any more complicated than Option 1.

The following is untested...might require a bit of debugging.

In both options, I would subclass CameraGroupStrategy to be used with the DecalBatch when drawing the mask decals, and override beforeGroups to also set the second texture.

public class MaskingGroupStrategy extends CameraGroupStrategy{

    private Texture fboTexture;

    //call this before using the DecalBatch for drawing mask decals
    public void setFBOTexture(Texture fboTexture){
        this.fboTexture = fboTexture;
    }

    @Override
    public void beforeGroups () {
        super.beforeGroups();
        fboTexture.bind(1);
        shader.setUniformi("u_fboTexture", 1);
        shader.setUniformf("u_screenDimensions", Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
    }
}

And in your shader, you can get the FBO texture color like this:

vec4 fboColor = texture2D(u_fboTexture, gl_FragCoord.xy/u_screenDimensions.xy);

Then for option 1:

gl_FragColor = vec4(fboColor.rgb, 1.0-texture2D(u_texture, v_texCoords).a);

or for option 2:

gl_FragColor = v_color * texture2D(u_texture, v_texCoords);
gl_FragColor.a *= fboColor.r;
Tenfour04
  • 83,111
  • 11
  • 94
  • 154
  • Hey it works perfecly on desktop! But on android nothings get drawn. Do you have some idea why? Ive tried with frame buffer format 8888 and 4444, but it's the same. – Veljko Jul 19 '14 at 10:54
  • I have debugged...drawing to frame buffer works, but step that uses custom group strategy, doesn't draw anything to the screen. This is my beforeGroupsMethod: Gdx.gl.glEnable(GL20.GL_DEPTH_TEST); shader.begin(); shader.setUniformMatrix("u_projTrans", camera.combined); Gdx.graphics.getGL20().glActiveTexture(GL20.GL_TEXTURE1); fboTexture.bind(1); shader.setUniformi("u_fboTexture", 1); shader.setUniformf("u_screenDimensions", Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); Gdx.graphics.getGL20().glActiveTexture(GL20.GL_TEXTURE0); shader.setUniformi("u_texture", 0); – Veljko Jul 19 '14 at 11:09
  • When something works on desktop but not Android, for me it is usually because of something in the shader causing the shader not to compile. After compiling your shader, put this for debugging purposes: `if (!shader.isCompiled())Gdx.app.log("shader compile failed", shader.getLog());`. This will tell you any warnings or errors that happened when compiling. It actually gives you the line number and column number where the errors occur, but not whether they happen in the vertex or fragment shader, so sometimes you have to check both. – Tenfour04 Jul 19 '14 at 18:37
  • Hey I've tried my code on the phone, it's working...also I've tried it on STB device, and there's nothing gets drawn. There are no error or warning log in shader. I've even tried to replace it to draw simple red box, but it makes no difference. Do you know some settings that might help? STB device is my target platform, so it has to work on it. Any suggestions? It's Jelly Bean 4.2.2. – Veljko Jul 21 '14 at 08:57
  • I found solution: Only thing that I needed to add is v_color in calulus for frag color in fragment shader. Event it's useless, I guess there something wrong with compiler version on STB or something like that. – Veljko Jul 21 '14 at 10:36
  • Usually it's not something wrong in the compiler, but rather that the other devices you tested on were more lenient than the OpenGL ES standard requires about syntax details. For example, most GPU's will automatically cast integers to floats if it can be inferred that it should be a float, but this isn't required by OpenGL ES so there are a few devices where using an integer in the wrong place will cause it not to work. – Tenfour04 Jul 21 '14 at 12:40
  • I had an idea for improving performance...right now `fboColor` is calculated with a dependent texture read. You could calculate the equivalent of `gl_FragCoord.xy/u_screenDimensions.xy` in the vertex shader and pass it as a varying. I think you would do this by using `gl_Position` after you've calculated it. So `v_fboTexCoords = gl_Position.xy * 0.5 + 0.5;`. – Tenfour04 Jul 21 '14 at 12:49
  • You optimization trick, works, but the output isn't the same...everything gets smaller in the mask. I don't really understand what are those constant factors that you multply and add. Could you explain the idea and purpose of it? – Veljko Jul 21 '14 at 13:36
  • When you multiply vertex position by your projection matrix to get `gl_Position` it is transforming the position to screen space where the screen is mapped from {-1,-1}(lower-left) to {1,1}(upper-right). So I was dividing by two and adding 0.5 to get it mapped to {0,0} to {1,1}. But maybe I made an error somewhere. – Tenfour04 Jul 21 '14 at 13:51
  • I understand the idea. Like I said it works, but it seems that masked texture is 2 times smaller then the texture with previous method. I can't fix it by moving my masked texture closer to the mask. But ok, if I had performance issues I will look closer to that. For now I will stick to the method 1. Tnx a lot for your help!!! – Veljko Jul 21 '14 at 14:07
  • Maybe just add 0.5 without multiplying by 0.5? – Tenfour04 Jul 21 '14 at 14:36