Strange bugs occur while using stencil testing

Question

I'm making an app which shows a terrain and some custom pits which should be inserted into the terrain. To insert the pits into the terrain I use stencil testing. It works almost great, but some strange bugs are present.

The code:

glEnable(GL_STENCIL_TEST);

glStencilFuncSeparate(GL_FRONT, GL_ALWAYS, 1, 255);
glStencilOpSeparate(GL_FRONT, GL_KEEP, GL_KEEP, GL_REPLACE);

glStencilFuncSeparate(GL_BACK, GL_ALWAYS, 1, 255);
glStencilOpSeparate(GL_BACK, GL_KEEP, GL_KEEP, GL_KEEP);

glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LEQUAL);
glDepthMask(GL_TRUE);

for (Pit3D* pit in pitList) {
    [pit render];
}

glStencilFuncSeparate(GL_FRONT, GL_EQUAL, 0, 255);
glStencilOpSeparate(GL_FRONT, GL_REPLACE, GL_KEEP, GL_KEEP);

glStencilFuncSeparate(GL_BACK, GL_EQUAL, 0, 255);
glStencilOpSeparate(GL_BACK, GL_REPLACE, GL_KEEP, GL_KEEP);

[terrain render];

Bug #1:

When I look through a hill to the pits I see some strange shadows or black spots. Although I look through an another hill to the pits I don't see that shadows or spots.

Bug #2:

Some strange pixels are present when the pits are close to the camera. They definitely belongs to the pits but they shouldn't be visible.

Unfortunately I don't have screenshot at the moment. It will be added later today.

Bug #3

Depends on range and view angle sometimes the pits are covered by the terrain. But when the pits are close to the camera the terrain cover is disappeared from the pits.

I have no idea how to fix these bugs. Please help me. All answers will be upvoted.

UPDATES:

I've recently discovered that the pits are covered by terrain back-face.

It's almost impossible to help within knowing exactly what you are doing. Please read https://stackoverflow.com/help/mcve and post a complete example which reproduces the problem. — solidpixel, Apr 12 '18 at 09:12
@solidpixel, please say me what do you need to know for better understanding the problem? Do you need the code of the [terrain render] and [pit render]? Or what exactly should I add to this question? — Dmitry, Apr 12 '18 at 14:59
@solidpixel, I've updated the code and added the place I call glEnable( GL_STENCIL_TEST ). Also I've discovered that the pits are covered by terrain back-face, please see updates. — Dmitry, Apr 12 '18 at 18:56

score 3 · Accepted Answer · answered Apr 14 '18 at 14:17

To achieve what you want you have either to draw the scene from the back to the front. Note, the back faces of the geometry have to clear the stencil buffer, before the font faces are drawn.

Or you draw the scene twice, using Face Culling. When you draw the scene first, then cull all the front faces and when you draw it second then cull all the back faces.
Use the separated stencil test for front and back faces in that way, that th back faces clear the stencil buffer. (see glStencilFuncSeparate and glStencilOpSeparate)

The rendering process works as follows:

Enable the depth test
Disable the color buffer and enable the stencil test for setting the stencil mask
Draw the "holes". This causes that the stencil buffer is set to 1 at the positions of the holes.
Setup the stencil test for clearing the stencil buffer
Enable face culling for front faces
Draw the geometry. This causes that the stencil buffer is cleared at the positions where the "holes" covered from the back faces of the geometry.
Enable face culling for back faces
Enable the color buffer
Draw the geometry

To make tha algorithm work, you have to draw all your primitives int the same winding order. And you have to tell OpenGL the direction by glFrontFace. Either clockwise GL_CW or counterclockwise GL_CCW.

glStencilMask(0xFF);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glFrontFace(GL_CCW);  // depends on your geometry "GL_CCW" or "GL_CW"
glDisable(GL_CULL_FACE);

glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LESS); // default

glColorMask(GL_FALSE,GL_FALSE,GL_FALSE,GL_FALSE);

glEnable(GL_STENCIL_TEST);

glStencilFuncSeparate(GL_FRONT, GL_ALWAYS, 1, 255);
glStencilFuncSeparate(GL_BACK, GL_ALWAYS, 1, 255);
glStencilOpSeparate(GL_FRONT, GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilOpSeparate(GL_BACK, GL_KEEP, GL_KEEP, GL_KEEP);

// draw the holes
// ....

glStencilFuncSeparate(GL_FRONT, GL_EQUAL, 0, 255);
glStencilOpSeparate(GL_FRONT, GL_KEEP, GL_KEEP, GL_KEEP);
glStencilFuncSeparate(GL_BACK, GL_ALWAYS, 0, 255);
glStencilOpSeparate(GL_BACK, GL_KEEP, GL_KEEP, GL_REPLACE);

glEnable(GL_CULL_FACE);
glCullFace(GL_FRONT);

// draw the geometry the 1. time ("draw" back faces)
// ....

glCullFace(GL_BACK);
glColorMask(GL_TRUE,GL_TRUE,GL_TRUE,GL_TRUE);

// draw the geometry the 2. time (draw front faces)
// ....

Strange bugs occur while using stencil testing

1 Answers1