How can I make a Stencil Buffer mask only where the mesh is drawn?

Question

I was trying to mask a sprite using a stencil buffer and it works fine untill I add some transparency to it. The mask works using the whole texture, but I want it to work only where the quad is drawn. Here's a picture to illustrate what I'm saying:

The black sprite is the sprite I want to mask and as you can see the quad that is acting as the mask isn't masking where there's transparency in the texture

I'm just passing Unity's default quad mesh to the Mesh Filter

Is this normal behaviour? Is there anything I can do to make it work only where the quad is being drawn?

The mask shader code:

    Tags { "RenderType" = "Transparent" }
        
        Pass
        {
            Stencil
            {
                Ref 1
                Comp Always
                Pass Replace
            }

            ColorMask 0 // Don't write to any colour channels
            ZWrite Off // Don't write to the Depth buffer

        }

The mask interaction code:

    Tags
        {
            "RenderType" = "Transparent"
            "Queue" = "Transparent"
        }

        Pass
        {
            Name "Sprite Lit"
            Tags
            {
                "LightMode" = "Universal2D"
            }

        // Render State
        Stencil
        {
            Ref 1
            Comp Equal
        }

        Blend SrcAlpha OneMinusSrcAlpha, One OneMinusSrcAlpha
        Cull Off
        ZTest LEqual
        ZWrite Off
    }

I think you have to use the `Blend` option in the Mask Shader. Try `Blend SrcAlpha OneMinusSrcAlpha` — KYL3R, Aug 29 '22 at 13:16
Wait, I'm a bit confused here. I'm a shader newbie, so if you can explain some things to me I'd appreciate. That "discard" that you wrote is a default function of HLSL or something like that? — Hemessfell, Aug 29 '22 at 14:14

score 1 · Accepted Answer · answered Aug 29 '22 at 13:20

I think you have to use the Blend option in the Mask Shader. Try Blend SrcAlpha OneMinusSrcAlpha - but I'm not sure if that's enough.

Next option would be to manually discard the transparent pixels in the fragment shader of your MaskShader (that writes the stencil mask):

vec4  textureRGBA = (sample texture here);
float alpha_threshold = 0.01;
if (textureRGBA.a < alpha_threshold)
    discard; // discard the pixel - will not be drawn and will not write stencil mask for this pixel.

How can I make a Stencil Buffer mask only where the mesh is drawn?

1 Answers1