How do I use the stencil buffer to mask only the object pixels that are BEHIND another object?

Question

I have two objects, we'll call them Mask and Masked. I want to use the shape of Mask to "cut a hole" in Masked. So far, I've been able to set up a very simple test, which renders as shown below: https://i.stack.imgur.com/Ct2ZR.png

Unfortunately, this mask doesn't take into account the Z depth of Mask.

What I'd like to see is something like this: https://i.stack.imgur.com/rxbL4.png

To clarify, Mask (the sphere) is intersecting Masked (the cube), but the full shape of the sphere is being used as the stencil, instead of just the pixels which are not obscured by the cube.

Here's my Stencil pass for the sphere:

    ColorMask 0
    ZWrite off
    Pass{
        Stencil {
            Ref 1
            Comp always
            Pass replace
        }

Here's my Stencil pass for the cube:

    Stencil {
        Ref 1
        Comp notequal
    }

Its seems the problem is that the sphere is not failing the depth test, which makes sense since it's first in the Render Queue ("Queue"="Geometry-1"). Any ideas on what I can try to make this work?

score 0 · Answer 1 · answered Jul 16 '15 at 12:24

Essentially, you need to cut the sphere with Z test from the cube, and you need to cut the cube with an stencil test from the sphere. This chicken-egg problem can probably be solved with another "pass" in your cube. Make it first write with ColorMask 0 in Queue "Geometry-2". Then, render your sphere as usual. And change the current cube pass to ZTest with Equal.

How do I use the stencil buffer to mask only the object pixels that are BEHIND another object?

1 Answers1