How does gl_FragColor affect the stencil buffer?

Question

I have set up my operations to draw to the stencil buffer, similar to the following:

void onDisplay() {
  glClear(GL_DEPTH_BUFFER_BIT);
  glEnable(GL_STENCIL_TEST);
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);
  glStencilFunc(GL_NEVER, 1, 0xFF);
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);

  // Draw to stencil buffer
  glStencilMask(0xFF);
  glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF
  draw_circle();

  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);
  glStencilMask(0x00);

  // draw where stencil's value is 0
  glStencilFunc(GL_EQUAL, 0, 0xFF);
  /* (nothing to draw) */
  // draw only where stencil's value is 1
  glStencilFunc(GL_EQUAL, 1, 0xFF);

  draw_scene();

  glDisable(GL_STENCIL_TEST);
}

Now, if I have the following fragment shader enabled when I call draw_circle() (above):

void main() {
  gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
}

How will the values of the stencil buffer differ from if I were to use the following fragment shader?

void main() {
  gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0);
}

In other words, I'm wondering how the output of the fragment shader affects the stencil buffer when drawing to the stencil buffer.

Edit:

The point of my question is to correct some misunderstandings I know I have regarding the stencil buffer. One example I have that I think explains the stencil buffer fairly well is [1]. Here, the following is mentioned:

The glColorMask function allows you to specify which data is written to the color buffer during a drawing operation. In this case you would want to disable all color channels (red, green, blue, alpha). Writing to the depth buffer needs to be disabled separately as well with glDepthMask, so that cube drawing operation won't be affected by leftover depth values of the rectangle. This is cleaner than simply clearing the depth buffer again later.

So, it seems from this page, that, in order to write to the stencil buffer, one needs to enable/disable the appropriate modes (i.e. color and depth), and then go through the entire rasterization process, which will only write to the stencil buffer. Since the rasterization process includes the fragment shader, is the output of the fragment shader (i.e. gl_FragColor) simply ignored? How can I tell GL what to write to the stencil buffer position (x, y)?

[1] : https://open.gl/depthstencils

score 3 · Accepted Answer · edited Jan 19 '17 at 19:38

3

Unless you have access to AMD/ARB_shader_stencil_export, the fragment shader cannot directly affect the output to the stencil buffer. The only exception to this is discarding the fragment.

And according to this database, only AMD cards provide this extension. Also, that extension exposes an output specifically for the stencil. It modifies the stencil value of the fragment; the color values of the fragment never affect the fragment's stencil value.

edited Jan 19 '17 at 19:38

jwir3

6,019
5
47
92

answered Jan 19 '17 at 17:57

Nicol Bolas

449,505
63
781
982

Uhm... I wonder if one can somehow render into a R8UI texture and read it back as a STENCIL_INDEX8. (Would require creative interpretation of the specification, I guess). – peppe Jan 19 '17 at 18:41
@peppe: No. You can't even copy (on the GPU) between color and stencil textures. You'd have to copy to CPU memory, then copy it back to GPU memory. – Nicol Bolas Jan 19 '17 at 18:43
Hm, I guess perhaps my question wasn't completely clear. In order to write to the stencil buffer, don't you need to go through the whole rasterization pipeline, including the fragment shader? So, I'm wondering if, while drawing to the stencil buffer, the output of the fragment shader (i.e. gl_FragColor) affects the stencil buffer? – jwir3 Jan 19 '17 at 19:37
@NicolBolas Perhaps this is a misunderstanding on my part of how the stencil buffer is written to. If I wanted to draw a circle to the stencil buffer, for example, but, say I wanted to write a value to the stencil buffer that depended on how far from the center of the circle each given fragment was, how would I go about doing this? – jwir3 Jan 19 '17 at 19:40
1

You can't do that in a straightforward way (without the extension mentioned above). Yes, you need the full rasterization pipeline to write in the stencil buffer, but what you can write in there is very limited and controlled only by `glStencilOp` -- basically, very simple integer arithmetic. Why do you need such a complex fill of the stencil buffer anyhow? – peppe Jan 19 '17 at 19:49
@peppe I don't need a complex fill of the stencil buffer. I'm simply trying to determine what _happens_ in the output of the fragment shader when it's written to the stencil buffer. For example, if I have a fragment shader that outputs `gl_FragColor = vec3(1.0, 0.0, 0.0)` and one that outputs `gl_FragColor = vec(1.0, 1.0, 0.0)`, are these going to write different values to the stencil buffer if used for that purpose, or is color _completely_ ignored? – jwir3 Jan 19 '17 at 19:55
Color (and other outputs in case of MRT) is completely ignored. What happens is determined exclusively by `glStencilOp(Separate)`, `glStencilFunc(Separate)`, if stencil testing is enabled, and if you've got a stencil buffer in the first place. – peppe Jan 19 '17 at 20:01
@peppe I see, so unless `discard` is hit in the fragment shader, the fragment will be written in the stencil buffer, and the value written is determined by those two methods? So, the fragment shader has two possible outputs for a stencil buffer: `discard`, which essentially writes 0 to the stencil buffer, or non-`discard`, which writes whatever value I specify? – jwir3 Jan 19 '17 at 20:05
A `discard`ed fragment does not touch the stencil buffer at all. A non-`discard`ed fragment *modifies* that fragment's value in the stencil buffer according to the operations set in `glStencilOpSeparate`. Those in turn depend on whether the fragment is front facing or back facing, if passes or fails the stencil test (as determined by `glStencilFuncSeparate`), and if it passes the stencil, if it passes or fails the depth test. (All of this assuming that the stencil test is active and you've got a stencil buffer) – peppe Jan 19 '17 at 20:13
Thank you, @peppe. If you post that as an answer, I will accept it. – jwir3 Jan 19 '17 at 20:30
@jwir3: I said exactly what peppe did. I don't know how I could be more clear: "the color values of the fragment ***never affect the fragment's stencil value***." Or "the fragment shader ***cannot directly affect the output to the stencil buffer***" – Nicol Bolas Jan 19 '17 at 20:33
1

@NicolBolas I understand. My question wasn't clear enough for the answer I desired. peppe talked me through what I actually wanted to know. I will uprank your answer. If I could accept two answers, I would, but I feel like peppe gave me more assistance in learning what it was I desired to learn. – jwir3 Jan 19 '17 at 20:36
Intel GPUs also support ARB_shader_stencil_export nowadays. Nvidia is still the holdout though. – Fizz Jan 29 '22 at 21:26

How does gl_FragColor affect the stencil buffer?

1 Answers1