1

I am planning to draw a rectangle with a hole in its center. I am trying to use the stencil test on it but I can't make it work. You can see below for how I do it.

glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE,GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 2, ~0);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
// Draw the rectangle here
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 2, ~0);
glStencilOp(GL_INCR, GL_INCR, GL_INCR);

What am I doing wrong here? Any help would much be appreciated! :)

zyneragetro
  • 139
  • 1
  • 2
  • 13
  • This just sets up writing `2` into the stencil buffer (assuming it's deep enough), then the "draw here" code which will write that `2` wherever it draws, and then sets up the stencil to only draw where the `2` was written. This doesn't match the textual description much. Can you show a more complete (pseudo-)code example? – Angew is no longer proud of SO Sep 28 '15 at 07:57
  • @Angew - This is the only code I have for the stencil test. I am new to this and I would really want to learn how it should be done. The "draw here" is just a simple binding of buffer to opengl and calling the glDrawArrays. There is nothing more I can give to you from this. – zyneragetro Sep 28 '15 at 08:07

1 Answers1

3

I assume your situation is that you already have something drawn in the framebuffer, and now you want to draw a rectangle with a hole in it, so that it does not cover what's underneath the hole, but covers what't underneath the non-hole part.

Logically, this means you first draw the hole into the stencil buffer, and then use the stencil test to exclude those fragments when drawing the rectangle.

In code, it looks something like this:

glEnable(GL_STENCIL_TEST);

// Fill stencil buffer with 0's
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);

// Write 1's into stencil buffer where the hole will be
glColorMask(GL_FALSE,GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawHoleShape();

// Draw rectangle, masking out fragments with 1's in the stencil buffer
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilFunc(GL_NOTEQUAL, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
drawRectangle();

// Cleanup, if necessary
glDisable(GL_STENCIL_TEST);

Of course, you can use 2 (or any other stencil bit/value) instead of 1.

Angew is no longer proud of SO
  • 167,307
  • 17
  • 350
  • 455
  • Thank you for the example. I will try this first :) – zyneragetro Sep 28 '15 at 08:37
  • Sorry for the late response. I have followed the algorithm above and it seems whatever I draw in the drawholeshape disappears and the only thing that I can see is still the whole rectangle. – zyneragetro Sep 29 '15 at 01:17
  • 2
    @zyneragetro Something just occurred to me - have you actually created a stencil buffer in your context? E.g. when using GLUT, it would be done by adding `GLUT_STENCIL` to the arguments of `glutInitDisplayMode`. – Angew is no longer proud of SO Oct 01 '15 at 08:03
  • Nope. What you have given to me above is the only thing I wrote. What happens when you create a stencil buffer? – zyneragetro Oct 05 '15 at 00:23
  • 1
    @zyneragetro You can use it. You can't write into a stencil buffer if you don't have one. – Angew is no longer proud of SO Oct 05 '15 at 06:33
  • Thanks but I can't seem to find what is the equivalent glutInitDisplayMode when not using GLUT. – zyneragetro Oct 05 '15 at 07:33
  • 1
    @zyneragetro OK then, *what* are you using instead of GLUT? BTW, please don't expect me to magically provide a 100% complete solution with all the t's crossed and i's dotted; that's not what SO is for. I am more than happy to help out, but I expect research and effort from your side as well. – Angew is no longer proud of SO Oct 05 '15 at 07:41
  • Wait. I am confused now but in my header I import this OpenGL/gl.h, I tried the glutInitDisplayMode but it won't work, I searched and can't seem to find anything yet and also no worries as I don't expect you to answer everything :) – zyneragetro Oct 05 '15 at 08:09
  • @zyneragetro GLUT is not part of GL itself, it's a small utlity library for managing related-but-not-GL stuff like window and context creation. What are you using to get a GL-capable window on screen? It is in *that* that you must somehow enable the stencil buffer. – Angew is no longer proud of SO Oct 05 '15 at 08:48
  • I am using EGL and until now I can't find how to enable the equivalent of GLUT_STENCIL. I am still researching and reading documentations about how to do it. – zyneragetro Oct 07 '15 at 02:04
  • @zyneragetro I've never used EGL, but browsing the [spec](https://www.khronos.org/registry/egl/specs/eglspec.1.5.pdf), it seems you should look into things like `eglChooseConfig` and `EGL_STENCIL_SIZE` (section 3.4). – Angew is no longer proud of SO Oct 07 '15 at 06:25
  • Thanks :) I will just update you once I found the right thing to used. Thank you for the patience and time :D – zyneragetro Oct 07 '15 at 09:11
  • 1
    Using the above answer you posted plus the information you have given about enabling the stencil buffer, my problem is solved! Thanks :) – zyneragetro Oct 08 '15 at 07:59