how to do customized stencil test in fragment shader

Question

In my project, I used 'discard' call to perform customized stencil test, which tries to draw things only on a specified area defined by a stencil texture. Here is the code from fragment shader:

//get the stencil value from a texture
float value=texture2D( stencilTexture, gl_FragCoord.xy/1024.0).x;
//check if value equals the desired value, if not draw nothing
if(abs(value-desiredValue)>0.1)
{
    discard;
}

This code works, but suffers from a performance problem because of the 'discard' call. Is there an alternative way to do this through GPU Shaders? Tell me how.

why cannot you use default stencil test (not in shaders)? you will get the same result. — fen, May 21 '13 at 10:03
"*alliterative*" I think you meant "alternative". Also, it's not clear why you need to use a *stencil* texture to do this. Or why it is you think that the `discard` call is a performance problem (or if it is a performance problem, that the problem is somehow worse than the texture access itself). — Nicol Bolas, May 21 '13 at 10:17
@NicolBolas: sorry for my misspelling, in my experiment I found that discard call is a performance problem, which is also reported by [link](http://stackoverflow.com/questions/8509051/is-discard-bad-for-program-performance-in-opengl) — Sunf71, May 21 '13 at 14:12
@fen: thanks for your reply. I want to use stencil test in shaders because I want to implement drawing my scene on different predefined areas in a single render pass. — Sunf71, May 21 '13 at 14:18
@Sunf71: Then it's not really a "stencil test"; it's just discarding a fragment based on a value fetched from a texture access. Stenciling is specifically about using values written by one rendered object (visible or not) to prevent writes to those rendered locations. — Nicol Bolas, May 21 '13 at 14:44
@Sunf71: "*in my experiment I found that discard call is a performance problem*" On what hardware? And are you sure you tested it correctly? Are you sure you're not just measuring the time it takes to access the "stencil" texture? Because if you remove the `discard`, and you don't use the fetched value anywhere, then most compilers will completely optimize out the texture access. — Nicol Bolas, May 21 '13 at 14:50

score 5 · Accepted Answer · edited May 23 '17 at 11:50

If you access a texture, you must suffer the performance penalties associated with accessing a texture. In the same way, if you want to stop a fragment from being rendered, you must suffer the performance penalties associated with stopping fragments from being rendered.

This will be true regardless of how you stop that fragment. Whether it's a true stencil test, your shader-based discard, or alpha testing, all of these will encounter the same general performance issues (for hardware where discard leads to any significant performance problems, which is mainly mobile hardware). The only exception is the depth test, and that's because of why certain hardware has problems with discard.

For platforms where discard has a substantial impact in performance, the rendering algorithm works most optimally if the hardware can assume that the depth is the final arbiter of whether a fragment will be rendered (and thus, the fragment with the highest/lowest depth always wins). Therefore, any method of culling the fragment other than the depth test will interfere with this optimization.

Thanks for your reply! Very reasonalble answer! – Sunf71 May 24 '13 at 07:59 — Sunf71, May 24 '13 at 07:59

how to do customized stencil test in fragment shader

1 Answers1