1

I want to have discard in my fragment shader prevent that fragment from writing to the stencil buffer; I had an idea for a visual effect in my game, and it might be implementable if I can rely on that behaviour for discard.

My fragment shader doesn't do anything very fancy; it just samples a texture, and if the alpha is near 0, it discard otherwise it writes.

void main()
{
    vec4 texColor = texture(tex, texCoord);
    if(texColor.a < 0.5f)
    {
        discard;
    }
    FragColor = texColor;
}

However, it sounds like I cannot rely on discard preventing the stencil buffer from getting written to:

If Stencil Testing is active, then discarded fragments can still affect the stencil buffer. The stencil test can modify the stencil buffer, even on a stencil or depth test failure. And since the stencil test happens before the depth test, the depth test failures cannot prevent the stencil test from updating the stencil buffer.

I do need the stencil test active while I'm rendering. Can I not rely on discard preventing writes to the stencil buffer?

Mr. Smith
  • 4,288
  • 7
  • 40
  • 82

1 Answers1

5

By the normal rules of OpenGL's order of operations, the stencil test happens after the execution of the fragment shader. As such, a fragment shader which executes a discard statement will prevent writes to framebuffer images. With one exception.

That exception being that the order of operations can be changed by explicit request of the FS. If no such request is given, then things must proceed as defined by the original order (which is why using discard turns off early fragment tests as an optimization).

The paragraph in question is talking about how part of the operation called the "stencil test" includes potentially updating to the stencil buffer. That is, the stencil test can both discard a fragment and change the stencil buffer's value based on how it was discarded. This is unlike the depth test, which if it fails, never updates the depth buffer's value.

Nicol Bolas
  • 449,505
  • 63
  • 781
  • 982
  • But that's also what I'm talking about; I need it to not write to the stencil buffer if I `discard` -- it's literally the title of my question. – Mr. Smith Aug 14 '20 at 06:21
  • @Mr.Smith: I'm not sure how I could be more clear without removing important information. If you *don't* explicitly force early fragment tests, then the fragment will be discarded before the stencil test because the the stencil test executes *after* the FS. What's in my third paragraph explains what the Wiki was talking about, but that only matters if you *get* to the stencil test. Which won't happen if you discard the fragment beforehand because... that's what discarding means. – Nicol Bolas Aug 14 '20 at 13:28
  • 1
    Ohhhhh; I understand now; thanks! Now if I could understand why Stackoverflow decided it was a good idea to prevent me from undo'ing my -1 >.> – Mr. Smith Aug 14 '20 at 18:41
  • 2
    @Mr.Smith: Downvotes after 5 minutes can only be retracted after an edit to the post (which I just did). This is done to prevent competitive downvoting, where you downvote a competing answer to make yours more prominent, then retract your downvote some time later to get back the -1. – Nicol Bolas Aug 14 '20 at 20:34