0

for a current 2D project I am rendering different objects on a scene. On top of this I render images which have a cut out part, for example a transparent circle on a black image. When moving the cut-out circle, this creates the effect that of course only the within the transparent part, the background objects are visible.

Now I want to add a second masking layer with a different transparent shape on it and create a union of these two, showing the background images underneath each of the transparent parts.

The following images show an example illustration:

Background objects Background objects

Mask 1 Masking image 1

Mask 2 Masking image 2

Desired Result Desired Result

For rendering, I am using libgdx with OpenGL 2.0 and scene2d as scenegraph. Basically, the background objects are added as actors onto a stage and then another Group-object rendering the masks.

Now I've tried by setting the Blending-function while rendering the masks, but I can't figure out if its possible to "unionize" the alpha values of each mask. But is that even possible?

I've though about using stencil buffers but I can't get this to work yet. I would be thankful if anybody could give me an approach on how to achieve this effect. Also, using stencil buffers would result in a pretty chopped of edge as the mask is either 0 or 1, correct?

Nicol Bolas
  • 449,505
  • 63
  • 781
  • 982
florianbaethge
  • 2,520
  • 3
  • 22
  • 29
  • 2
    Stencil has a whole byte as far as I know so it can be from 1 to 255. But in your case I would say you should try to draw this with shader directly. If that is not possible then I would say try to draw to alpha channel first. You can disable color by using `glColorMask(0,0,0,1)` then draw your shapes to it. Next draw your image blending with destination alpha `glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA)`. – Matic Oblak Jul 16 '19 at 07:08
  • So you would only render the masking shapes to the alpha channel in the color mask, so that within the circles, the alpha mask is opaque and outside transparent? And then the shapes on top with it, but the alpha channel would then only show the parts inside the masking circles? – florianbaethge Jul 16 '19 at 07:20
  • 1
    Exactly, yes. It does look like what you are trying to do. – Matic Oblak Jul 16 '19 at 07:22

1 Answers1

1

A potential approach could be to use render-to-texture and compositing manually. I'm saying "potential", because there's hardly one best way here. Using built-in blending modes can certainly have some performance gains, but it limits you to the provided blending function parameters. While certainly doable with stuff like rendering the mask to the framebuffer alpha channel, and then using that with GL_DST_ALPHA/GL_ONE_MINUS_DST_ALPHA, it gets tricky once your layout gets more complex.

Render-to-texture, OTOH, has no such drawback. You're taking the control of the entire compositing function and have the freedom to do whatever processing you wish. To elaborate a bit, the rendering would work like this:

  • Set up a texture for the objects, and render your objects to it.
  • Set up a texture for the mask - this could be e.g. one-channel 8-bit. Retarget the rendering to it, and render the mask with a shader that outputs the mask value.
  • If you want to add another mask, you can either render more stuff to the same mask texture, or create yet another one.
  • Crucially, it doesn't matter which order the above operations are done, because they're completely separate and don't impact each other; in fact, if the mask doesn't change, you don't even need to re-render it.
  • Render a full-screen quad with your compositing shader, taking those two textures as inputs (uniforms).

So, to sum up, render-to-texture is a bit more flexible in terms of the compositing operation, gives you a way to do other post-effects like motiong blur, and gives you more leeway in the order of operations. OTOH, it imposes a certain limit on the number of textures or passes, uses more memory (since you'll be keeping the intermediate textures in, as opposed to just working on one framebuffer), and might have a performance penalty.

If you decide to stick to the built-in blending, it gets a bit trickier. Typically you'll want to have alpha 0 as "no image", and 1 as "all image", but in this case it might be better to think about it as a mask, where 0 is "no mask" and 1 is "full mask". Then, the blend func for the mask could simply be GL_ONE/GL_ONE, and for the final image GL_ZERO/GL_ONE_MINUS_DST_ALPHA. That certainly restricts your ability to actually do blending and masking at the same time.

There exists a function called glBlendFuncSeparate that might make it a bit more flexible, but that's still not gonna give you as many possibilities as the method mentioned above.

Alternatively, actually learning how to set up stencil buffer would solve that specific issue, since the stencil buffer is made with specifically this use in mind. There's a lot of tutorials online, but basically it amounts to a few calls of glStencil(Op|Func|Mask), optionally with disabling the writes to the color buffer with glColorMask.

Bartek Banachewicz
  • 38,596
  • 7
  • 91
  • 135