OperGL stencil buffer not acting how I expect

Question

I have a simple UI widget system and I'm using the stencil buffer to act as a clipper so that children of a widget can't be drawn outside the widget. Basically, the stencil value for everything is inside the bounds for this widget is incremented. Then, anything drawn after I make the clipper must be within the box.

The clipper constructor looks like this:

// Stencil buffer and ClipperStack.size() start at 0
// Increment any pixel in the rect on my current recursion level 
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
ClipperStack.push_back(Rect);
// only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
    glVertex2f(Rect.Left, Rect.Top);
    glVertex2f(Rect.Left, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Top);
glEnd();
// Stencil clipper drawn, 
glColorMask(1, 1, 1, 1);
glStencilMask(0);
// now only draw stuff that's that has the right clipper value
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

When the clipper goes out of scope the destructor runs, which looks like this:

// Decrement anything we previously incremented
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);
// Get the old rect
sf::FloatRect Rect = clipperStack.back();
ClipperStack.pop_back();
// Only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
    glVertex2f(Rect.Left, Rect.Top);
    glVertex2f(Rect.Left, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Top);
glEnd();
// now draw on regular color buffer again, 
// stencil buffer should be the same as before constructor call
glColorMask(1, 1, 1, 1);
glStencilMask(0);
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

However, when I run this only the direct children of my root widget are drawn. The children of children aren't drawn at all. I've tried a bunch of variations of this and I keep doing something wrong. I don't know where I'm going wrong with this.

score 1 · Accepted Answer · answered Mar 10 '11 at 08:27

1

In both the constructor and destructor I think you need to set the glStencilMask() to set every bit of the stencil buffer. E.g. if you have an 8-bit stencil buffer you want to use glStencilMask(0xFF);

Of course, if you've only got a 1-bit stencil buffer your code won't work at all, since you're trying to increment the stencil value for each level of sub-widget.

answered Mar 10 '11 at 08:27

Jackson Pope

14,520
6
56
80

The stencil buffer mask should default to all 1's. In any case, I tried this and it made no change. :( – Alex Mar 10 '11 at 09:38
@Alex, It will default to all ones, but you then overwrite it with 00000001 (for a 8 bit buffer). But regardless, if that doesn't solve the problem there must be something else wrong. – Jackson Pope Mar 10 '11 at 09:48
@Alex: How many bits does your stencil buffer have? How many do you ask for, and do you check for errors afterwards? – Jackson Pope Mar 10 '11 at 09:55
Just because i have a similar issue and don't want to instantly open a new question: Is the stencil INC increasing it by 1 or increasing the bits by one? Because i can only have 8 cascaded widgets instead of 256? – Felix K. Sep 15 '14 at 17:00
@Felix: The stencil INCR will increase the value of the stencil buffer at that location by one. – Jackson Pope Sep 15 '14 at 20:06

OperGL stencil buffer not acting how I expect

1 Answers1