Stencil buffers seem to not work properly

Question

I am writing an SDL2/modern OpenGL application that uses stencil buffers. I have written the following code in my renderer:

glEnable(GL_STENCIL_FUNC);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glStencilFunc(GL_NEVER, 1, 0xFF);  //Always fail the stencil test
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);   //Set the pixels which failed to 1
glStencilMask(0xFF);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

//Drawing small rectangle here

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

glStencilFunc(GL_EQUAL, 1, 0xFF);   //Only pass the stencil test where the pixel is 1 in the stencil buffer
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);   //Dont change the stencil buffer any further

//Drawing big rectangle here

glDisable(GL_STENCIL_FUNC);

The goal of the code above is to only draw the part of the big rectangle that fits in the small rectangle. Unfortunately, when I run the code the opposite happens, it renders the big rectangle with a hole in it the size of the small rectangle.

I have tried many more stencil functions, but they all result in the same, and this seems like it should work. So, does anybody have any ideas or can tell me where I am going wrong?

I apperantly don't have the reputation to embed pictures in my post but:

Intended result: https://i.stack.imgur.com/YuBIx.jpg Actual result: https://i.stack.imgur.com/GSrQK.jpg

Answer 1

The call glEnable(GL_STENCIL_FUNC); is just wrong, the correct enum is GL_STENCIL_TEST. So your code doesn't use the stencil buffer at all.

I can only guess why you get the result you got, then: Your code might draw the first rectangle into the depth buffer, so when you draw the second one, the fragments in that area might fail the depth test. So even when you correctly enable the stencil test, you still have to take care about the depht buffer here.