1

I want to be able to clip my postprocessing image passes to specific regions so that effects such as a blur would only affect theses regions

In order to do that i use the stencil buffer and my pipeline is as follows :

  1. Render some objects to the stencil buffer only, writing 1s
  2. Render some objects where the stencil value equals 1 (this works)
  3. Render some objects whatever there is in the stencil buffer
  4. Run postprocess passes (by drawing a quad with the image resulting of the 3 previous step as a bound texture) where the stencil value equals 1 (or always, depending on an attribute of my effects)

The results i get :

  • Black image when postprocess involves stencil buffer
  • 'Good' image when it does not
  • An image with non null values only outside the masks when i change `glStencilFunc(GL_EQUAL, 1, 0xFF);` to `glStencilFunc(GL_NOTEQUAL, 1, 0xFF);`

What strikes me is the fact that the image obtained with glStencilFunc(GL_ALWAYS, 1, 0xFF); is not even equal to the union of the other two : the one with glStencilFunc(GL_EQUAL, 1, 0xFF); is all black.

What is wrong with this code ?

gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, objectsTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);

// =================    Masks    ===================
gl->glEnable(GL_STENCIL_TEST);
gl->glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color buffer writing
gl->glStencilFunc(GL_ALWAYS, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
gl->glStencilMask(MASKSBITPLANE); // Write values as is in the stencil buffer
gl->glClear(GL_STENCIL_BUFFER_BIT);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskObjects()){
     renderBlankSceneObject(scobjptr, gl, glext);
}

// =================    Masked   ===================
gl->glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color buffer writing
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskedObjects()){
     renderSceneObject(scobjptr, gl, glext);
}

// ================= Raw objects ===================
gl->glDisable(GL_STENCIL_TEST);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getRawObjects()){
     renderSceneObject(scobjptr, gl, glext);
}

// ================= Postprocess ===================
auto& shaderEffects(renderGroup->shaderEffects());
if (renderGroup->areShaderEffectsMasked()){
     gl->glEnable(GL_STENCIL_TEST);
     gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
     gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
     gl->glStencilMask(0x00); // Disable writing to the stencil buffer
}
for (auto it(shaderEffects.begin()); it != shaderEffects.end(); ++it)
{
     gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, currentImageTex, 0);
     gl->glClear(GL_COLOR_BUFFER_BIT);

     // postprocess
     gl->glUseProgram(shaderEffect->program().programId());
     gl->glUniform1f(shaderEffect->m_timeLocation, m_time.elapsed());
     gl->glActiveTexture(GL_TEXTURE0 + GLShaderEffect::PROCESSED_IMAGE_TEXTURE);
     gl->glBindTexture(GL_TEXTURE_2D, processedTexture);
     // some glUniform* calls
     updateUniforms(gl, shaderEffect->ressourceClientsCollection());
     // some glActiveTexture + glBindTexture calls
     bindTextures(gl, shaderEffect->ressourceClientsCollection());
     glext->glBindVertexArray(shaderEffect->vao());
     gl->glDrawElements(GL_TRIANGLES, shaderEffect->elementsCount(), GL_UNSIGNED_INT, nullptr);

     swap(currentImageTex, objectsTex);
}
Adrien Grosjean
  • 53
  • 7
  • So what is the actual code for `postprocess`? It is hard to huess what the actual GL states are going to be in your code. – derhass May 07 '19 at 18:38
  • Please check my edited code – Adrien Grosjean May 09 '19 at 08:24
  • Can't seem to find the problem... Gonna keep looking a while longer... Two notes about how you use OpenGL: 1. To turn on/off stenciling, do `glEnable/glDisable(GL_STENCIL_TEST)` instead of stencil func/op/mask combo (such as the Raw objects section). 2. Since you only use 1 bitplane you might as well use 0x1 instead of 0xFF, preferably held in a variable called `something_plane`. Should you start using multiple planes it will be a life saver. 3. If you insist on adressing all planes, the right way to do it is `GLuint(-1)`, not `0xFF` – Andreas May 09 '19 at 10:10
  • What does the stencil look like before and after `glFramebufferTexture2D`? – Andreas May 09 '19 at 10:25
  • @Andreas Thank you for the advice :) _"What does the stencil look like before and after glFramebufferTexture2D?"_ Hard to tell because of the behavior i observe – Adrien Grosjean May 09 '19 at 17:12
  • Changing `gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);` to `gl->glStencilFunc(GL_NOTEQUAL, 1, MASKSBITPLANE);` gives this image : https://pasteboard.co/IdVFczs.jpg : Stencil value appears to be one in the stenciled regions. Yet changing back to EQUAL gives black image. Commenting the whole if (areShaderEffectsMasked) gives a good image, but if i also comment glDisable(GL_STENCIL_TEST), the image gets black again.. – Adrien Grosjean May 09 '19 at 17:27
  • I fear the code is still not complete enough to diagnose the issue. Also, what value is `MASKSBITPLANE`? It _could_ totally explain the asymmetry you're observing. – derhass May 09 '19 at 18:07
  • @derhass Thanks for joining us in the search. If edited according to my comments it is 0x1, or possibly 0xFF. Either cover the first plane. – Andreas May 09 '19 at 18:38
  • 1
    I don't see any obvious issue with the code you pasted - I do suspect that the issue is actually elsewhere. This shouldn't be hard to find using some GL debugger like Renderdoc, apitrace, or the GPU vendor's ones. – derhass May 09 '19 at 18:42
  • @AdrienGrosjean Oh dear, you are not using a graphics debugger… Never try program OpenGL without a graphics debugger! [Here are some options](https://www.khronos.org/opengl/wiki/Debugging_Tools). Step through each draw call and monitor the stencil buffer! – Andreas May 09 '19 at 18:44
  • Thanks to both of you ! I solved the problem (see edited post).. i feel dumb ^^ BTW Renderdoc is the bomb as would Jesse Pinkman say, this is going to change my life ! – Adrien Grosjean May 10 '19 at 08:54
  • 1
    @AdrienGrosjean: Please write an answer rather than editing it into your question. – Davis Herring May 10 '19 at 20:44

1 Answers1

1

The answer : I didn't restore the context after drawing, e.g. disable stencil testing at the end of my render pass. I use Qt to blit my framebuffer into a widget and the stencil test was still active with another stencil buffer attached : that is why the screen got black.

Conclusion : Always restore the context to its previous state when you use a framework

Adrien Grosjean
  • 53
  • 7