Best approach to draw clipped UI elements in OpenGL

Question

I've a complex UI-system which allows a lot of stuff which also can be done with WPF but supports multiple plattforms ( iOS, Android, Windows, ... ). It's not completed yet and now i'm facing the following issue:

My designer wants rotating objects! Rotating objects are far more complex than simple axis aligned ones, which are the reason i can't use glScissor. A little graphic which might help to understand the problem:

Effect

You can see that i need to clip the object "Subcontainer" by the bounds of the "Parent Container". As far as i know there are few options:

Use the stencil buffer, in this case i got a problem because i have objects which are not visible and must also influence the stencil buffer because they are might mask the child object. Also i have to draw each object twice because i need to decrease the stencil buffer when going back in hierarchy.
Cut the plane ( triangulate; or any other ui model ) which is used to draw the UI-object, this seems to be a lot of afford because they might clipped at different points ( imagine a container in a rotated container in a rotated container... ) and also it's really hard to clip them correctly and it might be a source of performance issues

However both seem to cause a lot of different issues and might be a source for performance leaks. Is there any other way to archive what i want or is there any way to improve the both approaches above?

Perhaps a screenshot of desired effect (in layers) will help to present the question better. — Kromster, Dec 06 '12 at 11:38
Btw , you can use polygon clipping libs like Clipper. Or opengl tesslib2.I would personally do it via stencil tests. — Michael IV, Dec 06 '12 at 12:14

score 5 · Accepted Answer · answered Mar 05 '13 at 00:11

I ended up with using the Stencil-Buffer, this generates more draw calls than the depth-approach but is much easier to implement.

Before i draw i wrote this code:

if (_Mask)
{
    if (Stage.StencilMaskDepth++ == 0)
        GL.Enable(EnableFlags.STENCIL_TEST);

    GL.ColorMask(false, false, false, false);
    GL.DepthMask(false);
    GL.StencilFunc(StencilFunction.ALWAYS, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.INCR, StencilOp.INCR, StencilOp.INCR);

    // Draw rectangle
    DrawColor(Colors.Black);

    GL.ColorMask(true, true, true, true);
    GL.DepthMask(true);
    GL.StencilFunc(StencilFunction.EQUAL, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.KEEP, StencilOp.KEEP, StencilOp.KEEP);
}

After all childs have been drawn this code is called:

if (_Mask)
{
    GL.ColorMask(false, false, false, false);
    GL.DepthMask(false);
    GL.StencilFunc(StencilFunction.ALWAYS, Stage.StencilMaskDepth, Stage.StencilMaskDepth);
    GL.StencilOp(StencilOp.DECR, StencilOp.DECR, StencilOp.DECR);
    // Draw rectangle
    DrawColor(Colors.Black);

    GL.ColorMask(true, true, true, true);
    GL.DepthMask(true);

    if (--Stage.StencilMaskDepth == 0)
        GL.Disable(EnableFlags.STENCIL_TEST);
}

Maybe i going to test some other approaches in a few month but currently this is the easiest to implement.

Kromster · Answer 2 · 2012-12-06T12:18:53.147

3

This is just an idea, but what about using depth buffer to do the masking?

Enable depth buffer and set glDepthFunc(GL_LEQUAL);
Render Container A and its frame at Z = 0
Render Container A internal area / background (where other nested containers will be) at Z = 1
Now you have a "depth stencil" with container frame at depth 0 and container internals at depth 1. That means that anything you render inbetween will be above internals, but below frame (and clipped by it)
Now with next Container B, render its frame to Z = 0.5 (it will get clipped by parent container A on GPU)
Render container B internal area at Z = 0.75
Now anything you want to render within container B will have to go at Z = 0.75. It will overlay containers internal area, but will be clipped by both container A and B frames.

edited Dec 06 '12 at 12:18

answered Dec 06 '12 at 11:46

Kromster

7,181
7
63
111

I don't think that this is working?! But maybe i don't understand your approach, could you explain it a little bit more? – Felix K. Dec 06 '12 at 11:53
@Krom Stern , would be nice to see a deeper explanation of your approach. – Michael IV Dec 06 '12 at 12:00
Now, after you explained your approach a little bit more, it seems to be a nice idea. Needs to be tested but when it's working its a much better approach than triangulation or using the stencil buffer. But needs some work to calculate the right z-values. – Felix K. Dec 06 '12 at 13:19
I'm not entirely sure about best Z values, but probably something along the lines "Each next Z is Z/2" could apply. Let me know how it works out! – Kromster Dec 06 '12 at 13:21

score 1 · Answer 3 · answered Dec 06 '12 at 12:53

1

Maybe you could try rendering to texture. Create a parent texture. Then render all children to that texture. Then render the parent texture to the screen deforming and displacing it as desired. This solution may or may not have issues, depending on what do you want to achieve. Especially if you animate the containers scale or have a very complex tree of many nested containers you might have performance issues.

answered Dec 06 '12 at 12:53

Eiver

2,594
2
22
36

Please describe how do you propose to render to texture? Each containers contents into separate one? When do rotations apply? (e.g. rotating texture will blur the picture in it) – Kromster Dec 06 '12 at 13:06
I also thought about this approach, but this seems to cause a lot of memory consumption and i don't think it can be used on mobile devices. – Felix K. Dec 06 '12 at 13:08
Any rotation will "blur" something. Each container the will have to have its own texture. My proposal is not perfect. Lots of containers means lots of render calls. Scaling is more problematic than rotation. If you choose your texture to be small, then you render it enlarged, then you will get those nasty pixels (similar to those when you do shadow mapping using texture and stencil in most games). To overcome this you must use large enough textures. But if you have lots of containers, it means that your app will be memory hungry. – Eiver Dec 06 '12 at 13:14
1

... Sorry, my solution does have issues. But it also has benefits. Like it is very easy to do some post-processing on the container. Not only rotation and displacement. You can even assign a custom fragment shader to that texture for additional effects. – Eiver Dec 06 '12 at 13:17
@Eiver That's correct, this is very simple and of course it's not a bad idea. But currently it's not applicable on mobile devices ( maybe it is in a few years ) because the graphic memory is too low. I thought about the possibility to render it into 3 textures which always have the same size, but this might have a bad impact on performance ( more texels to fill ). – Felix K. Dec 06 '12 at 14:11
Well... If you find the optimal solution, then don't forget to post it in this thread. I would be very interested to see how to best solve this problem, because the solution would solve many more common problems, not only the GUI controls rendering. – Eiver Dec 10 '12 at 09:17

Best approach to draw clipped UI elements in OpenGL

3 Answers3

Linked