Depth test not working correctly with stencil test

Question

I need to cut holes in the 3D surface using stencil buffer. Currently, everything works as expected, but the main problem is that holes are also visible through hills. How to prevent this behavior and hide holes if they are behind hills?

Current code:

        GLES20.glColorMask(false,false,false,false);
        GLES20.glDepthMask(false);
        GLES20.glEnable(GLES20.GL_STENCIL_TEST);
        GLES20.glStencilFunc(GLES20.GL_ALWAYS, 1, 0xFF);
        GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
        GLES20.glStencilMask(0xFF);
        GLES20.glClear(GLES20.GL_STENCIL_BUFFER_BIT);
        //drawing holes
        GLES20.glStencilFunc(GLES20.GL_EQUAL, 0, 0xFF);
        GLES20.glStencilMask(0x00);
        GLES20.glColorMask(true,true,true,true);
        GLES20.glDepthMask(true);
        //draw surface with elevation

@Rabbid76 depth func is GLES20.glDepthFunc(GLES20.GL_LEQUAL); — Ololoking, Apr 10 '18 at 16:01
@Rabbid76 I have elevation map, which is built by some coordinates, also I have coordinates(polygons) of pits. Now I trying to draw this pits correctly with elevation map. Elevation map without pits with enabled depth test works correctly. So pit can be anywhere on the map, on hill or plane. — Ololoking, Apr 10 '18 at 20:05
@Rabbid76 I have triangulated polygon. So it's can have random geometry. — Ololoking, Apr 10 '18 at 20:23
@Rabbid76 Under the hole I have another polygon on "zero" height, which renders before all manipulates with pits, and it's visible through the hole. — Ololoking, Apr 10 '18 at 20:35

Rabbid76 · Accepted Answer · 2018-04-14T14:10:37.857

A solution which provides the effect, can be achieved by Face Culling by a separated stencil test for front and back faces and (see glStencilFuncSeparate and glStencilOpSeparate). Sadly the back faces of the geometry have to be drawn in a separated step.

The process can be described in the following steps:

Enable the depth test
Disable the color buffer and enable the stencil test for setting the stencil mask
Draw the "holes". This causes that the stencil buffer is set to 1 at the positions of the holes.
Setup the stencil test for clearing the the stencil mask by backfaces
Enable face culling for front faces
Draw the geometry. This causes that the stencil buffer is cleared at positions which are covered.
Enable face culling for back faces
Enable the color buffer
Draw the geometry

To make tha algorithm work, you have to draw all your primitives int the same winding order. And you have to tell OpenGL the direction by glFrontFace. Either clockwise GL_CW or counterclockwise GL_CCW.

GLES20.glStencilMask(0xFF);
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);

GLES20.glFrontFace(GLES20.GL_CCW);  // depends on your geometry "GL_CCW" or "GL_CW"
GLES20.glDisable(GLES20.GL_CULL_FACE);

GLES20.glEnable(GLES20.GL_DEPTH_TEST);
GLES20.glDepthFunc(GLES20.GL_LESS); // default

GLES20.glColorMask(false,false,false,false);

GLES20.glEnable(GLES20.GL_STENCIL_TEST);

GLES20.glStencilFuncSeparate(GLES20.GL_FRONT, GLES20.GL_ALWAYS, 1, 0xFF);
GLES20.glStencilFuncSeparate(GLES20.GL_BACK, GLES20.GL_ALWAYS, 1, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_FRONT, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
GLES20.glStencilOpSeparate(GLES20.GL_BACK, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);

// draw the holes
// ....

GLES20.glStencilFuncSeparate(GLES20.GL_FRONT, GLES20.GL_EQUAL, 0, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_FRONT, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);
GLES20.glStencilFuncSeparate(GLES20.GL_BACK, GLES20.GL_ALWAYS, 0, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_BACK, GLES20.GL_KEEP, GLES20.GL_KEEP, GL_REPLACE);

GLES20.glEnable(GLES20.GL_CULL_FACE);
GLES20.glCullFace(GLES20.GL_FRONT);

// draw the geometry the 1. time ("draw" back faces)
// ....

GLES20.glCullFace(GLES20.GL_BACK);
GLES20.glColorMask(true,true,true,true);

// draw the geometry the 2. time (draw front faces)
// ....

Tried your solution, but can't make it work. Holes are not created. — Ololoking, Apr 10 '18 at 19:01
Holes are polygons which looks like a pit, where top vertices have the same "z" as geometry and the other vertices are deeper. My mission is to inject this pits into the geometry, — Ololoking, Apr 10 '18 at 19:30
I have elevation map, which is built by some coordinates, also I have coordinates(polygons) of pits. Now I trying to draw this pits correctly with elevation map. Elevation without pits with enabled depth test works correctly. So pit can be anywhere on the map, on hill or plane. — Ololoking, Apr 10 '18 at 20:03

Depth test not working correctly with stencil test

1 Answers1