How to use stencil buffer to achieve hierarchical clipping

Question

What I'm trying to achieve is illustrated on the image below:

Let's say we have stencil buffer in a state so only red section is filled at the moment. What actions do I need to perform when I update stencil buffer with a section which is marked yellow, so in the end only green section would be the final state of the stencil buffer?

I need this to achieve nested element content clipping, to prevent content of the element to be rendered beyond the boundaries of them both combined.

So far I have tried various boolean operations involving stencil test to no avail, which brought more confusion than any progress.

Please note that scissor test is not actual for this task, because elements may have arbitrary shape and rotation.

Stencil has 8 separate bit planes. You can associate the red with one plane and yellow with the other, and then check if both planes' bits are set; that's where they intersect. See opengl spec for glStencilFunc and glStencilOp for details, do tell if you don't get it after reading. — Andreas, Jun 17 '19 at 17:57
Another solution would be to pass stencil test if less than 2 and then increment value. 0 = no element, 1 = one element, 2 = overlapping elements — Andreas, Jun 17 '19 at 18:00
@Andreas the problem is that there might be multiple elements that share one parent, and the hierarchy can potentially go on forever in depth. — , Jun 17 '19 at 18:13
*"so in the end only green section would be the final state of the stencil buffer"* - this is unclear - what does state mean? The last comment makes it completely confusing. Can you formulate a rule, which clearly defines the fragments which should be identified in the stencil buffer? Do you mean all fragmens on which is draw more than once. Or do you mean the fragments on which is drawn in the first and in the second pass? — Rabbid76, Jun 17 '19 at 18:21
@Rabbid76, I'm trying to solve a problem regarding content clipping. Imagine that red rectangle is parent and yellow is child. Child is supposed to be contained inside parent, being partially hidden inside due to offset (or "scroll", if you will). Green rectangle is what supposed to be rendered inside child element. State means what is currently written to the stencil buffer. — , Jun 17 '19 at 18:26
@Ocelot I suppose a child can have a child - so you've layers. Each child is only allowed to draw in the region which is defined by its parent. So each layer has to count up the stencil buffer (`GL_INCR`) and is only allowed to draw, if the current value of the stencil buffer is equal `layer-1` — Rabbid76, Jun 17 '19 at 18:29
@Rabbid76, that might work, but will definitely introduce tree depth limit. I guess there is no other way then. — , Jun 17 '19 at 18:34
@Rabbid76, yep I'm aware of that, you can post the answer now. — , Jun 17 '19 at 18:37

Rabbid76 · Accepted Answer · 2019-06-17T19:03:36.097

If you've a hierarchical structure,then you've layers. Each child is only allowed to draw in the region which is defined by its parent. So each layer has to count up the stencil buffer (GL_INCR) and is only allowed to draw, if the current value of the stencil buffer is equal to the layer "depth":

int layer = 0;
glStencilFunc(GL_ALWAYS, 1, 0xff);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

// draw initial layer
// ...

layer ++;
glStencilFunc(GL_EQUAL, layer, 0xff);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);

// draw layer 1
// ...

layer ++;
glStencilFunc(GL_EQUAL, layer, 0xff);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);

// draw layer 2
// ...

Note, in general the stencil buffer has 8 bits, this limits the number of layers to 256.

How to use stencil buffer to achieve hierarchical clipping

1 Answers1