opengl stencil buffer - not quite got it working

Question

currently im drawing a background, then a transparent triangle into the stencil buffer, then a blue square where the triangle isnt being drawn. i was hoping the effect would be a blue square with a trianglular hole exposing the background. but all im getting is a black triangle drawn over everything else. i think im getting the desired effect on the square but the stencil is also being applied to the background.

my question is how can i adjust the following code to allow the background to be displayed through a triangular hole in the square?

static void Draw(void)
{
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

//big blue square
glColor3ub(0,255,255);
glBegin(GL_POLYGON);
glVertex3i(5, 5, 0);
glVertex3i(-5, 5, 0);
glVertex3i(-5, -5, 0);
glVertex3i(5, -5, 0);
glEnd();

glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

/* transparent triangle */
glColor4ub(0.0f, 0.0f, 0.0f, 0.0f);
glBegin(GL_POLYGON);
glVertex3i(-4, -4, 0);
glVertex3i(4, -4, 0);
glVertex3i(0, 4, 0);
glEnd();

glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_INCR, GL_KEEP, GL_DECR);

glStencilFunc(GL_EQUAL, 2, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

/* blue square */
glColor3ub(0, 0, 200);
glBegin(GL_POLYGON);
glVertex3i(3, 3, 0);
glVertex3i(-3, 3, 0);
glVertex3i(-3, -3, 0);
glVertex3i(3, -3, 0);
glEnd();

if (doubleBuffer) {
    glutSwapBuffers();
} else {
    glFlush();
}
}

datenwolf · Accepted Answer · 2012-01-15T01:02:16.113

5

Problem one: When drawing the background either disable stencil test before, or switch the stencil function to GL_ALWAYS so that the fragments pass (I recomment turning it off). Then you're setting two different stencil modi afterwards, but neither will do, what you intend. So here's the change:

glDisable(GL_STENCIL_TEST);
glStencilMask(0x0);

draw_background();

glEnable(GL_STENCIL_TEST);
glStencilMask(0x1);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);

draw_trangle();

glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);

draw_blue_square();

edited Jan 15 '12 at 01:02

answered Jan 14 '12 at 11:32

datenwolf

159,371
13
185
298

thanks but that doesnt quite work. all im seeing is the square over the background with no triangular hole. could there be something about the way im setting things up in main? – cool mr croc Jan 14 '12 at 12:47
@westr: Yes, could be. I also admit, that I didn't test that code. Could you put your whole sourcecode at pastebin. That allows me to a) reproduce your problem and b) fix it in place. – datenwolf Jan 14 '12 at 14:12
1

been playing with it all day, finally got it working using my original code but with enabling alpha blending before drawing, i spose i should have been doing that from the start. if anyone else runs into the same problem or @datenwolf if im still doing it wrong, the working code is here http://pastebin.com/4xbMbwE9 – cool mr croc Jan 14 '12 at 23:10
1

I gave you a slightly wrong recipe, you need to have enabled the stencil test when drawing the masking triangle and set the stencil mask to include the bits you want to test for for the dark blue quad. See this version http://pastebin.com/vJFumvQz and my andjusted answer. – datenwolf Jan 15 '12 at 01:01

opengl stencil buffer - not quite got it working

1 Answers1

Linked