How to render a mesh behind another mesh, like a mask?

Question

I would like it so that when mesh A (the character), is behind mesh B (a wall), it is still rendered but with a solid gray color.

I'm beginning opengles 2.0 and I'm still unsure as to go about this. From what I understand the depth buffer allows meshes to fight out who will be seen in the fragments they encompass, also there are various blend functions that could possibly involved in this, finally the stencil buffer looks like it would also have this desirable functionality.

So is there a way to output different colors through the shader based on a failed depth test? Is there a way to do this through blending? Or must I use the stencil buffer some how?

And what is this technique called for future reference? I've seen it used in a lot of video games.

Answer 1

This can be done using the stencil buffer. The stencil buffer gives each pixel some additional bits which can be used as a bitmask or a counter. In your case you'd configure the stencil test unit to set a specific bitmask when the depth test for the character fails (because it's obstructed by the well). Then you switch the stencil test mode operation to pass the stencil test for this specific bitmask, and render a full viewport, solid quad in the desired color, with depth testing and depth writes disabled.

Code

I strongly recommend you dive deep into the documentation for the stencil test unit. It's a very powerful mechanism, often overlooked. Your particular problem would be solved by the following. I stuggest you take this example code, read it in parallel to the stencil test functions references glStencilFunc, glStencilOp.

You must add a stencil buffer to your frame buffer's pixel format – how you do that is platform dependent. For example, if you're using GLUT, then you'd add |GLUT_STENCIL to the format bitmask of glutInitDisplayMode; on iOS you'd set a property on your GLKView; etc. Once you've added a stencil buffer, you should clear it along with your other render buffers by adding |GL_STENCIL_BUFFER_BIT to the initial glClear call of each drawing.

GLint const silouhette_stencil_mask = 0x1;

void display()
{
    /* ... */    

    glEnable(GL_DEPTH_TEST);
    glDepthMask(GL_TRUE);
    glDepthFunc(GL_LESS);

    glDisable(GL_STENCIL_TEST);
    /* The following two are not necessary according to specification.
     * But drivers can be buggy and this makes sure we don't run into
     * issues caused by not wanting to change the stencil buffer, but
     * it happening anyway due to a buggy driver.
     */
    glStencilFunc(GL_NEVER, 0, 0);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

    draw_the_wall();

    glEnable(GL_STENCIL_TEST);
    glStencilFunc(GL_ALWAYS, silouhette_stencil_mask, 0xffffffff);
    glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP);

    draw_the_character();

    glStencilFunc(GL_EQUAL, silouhette_stencil_mask, 0xffffffff);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

    glDisable(GL_DEPTH_TEST);
    glDepthMask(GL_FALSE);

    draw_full_viewport_solid_color();

    /* ... */
}