First I'd like to mention that I found what I believe to be the exact same question, unfortunately without an answer, here: Java Using OpenGL Stencil to create Outline
I will post my code below, but first here is the problem: from this capture**, you can see that the entire frame structure is showing, instead of a single line around the sphere. I would like to get rid of all those lines inside!
** Apparently I cannot add pictures: see this link - imagine a sphere with all the edges of the quads visible in big 3 pixels large lines.
http://srbwks36224-03.engin.umich.edu/kdi/images/gs_sphere_with_frame.jpg
Here is the code giving that result:
// First render the sphere:
// inside "show" is all the code to display a textured sphere
// looking like earth
sphe->show();
// Now get ready for stencil buffer drawing pass:
// 1. Clear and initialize it
// 2. Activate stencil buffer
// 3. On the first rendering pass, we want to "SUCCEED ALWAYS"
// and write a "1" into the stencil buffer accordingly
// 4. We don't need to actually render the object, hence disabling RGB mask
glClearStencil(0); //Edit: swapped this line and below
glClear(GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_NEVER, 0x1, 0x1); //Edit: GL_ALWAYS
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); //Edit: GL_KEEP, GL_KEEP, GL_REPLACE
glPolygonMode(GL_FRONT_AND_BACK, GL_FILL);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE); // As per Andon's comment
sphe->show();
// At this point, I expect to have "1" on the entire
// area covered by the sphere, so...
// 1. Stencil test should fail for anything, but 0 value
// RM: commented is another option that should work too I believe
// 2. The stencil op instruction at the point is somewhat irrelevant
// (if my understanding is correct), because we won't do anything
// else with the stencil buffer after that.
// 3. Re-enable RGB mask, because we want to draw this time
// 4. Switch to LINE drawing instead of FILL and
// 5. set a bigger line width, so it will exceed the model boundaries.
// We do want this, otherwise the line would not show
// 6. Don't mind the "uniform" setting instruction, this is so
// that my shader knows it should draw in plain color
// 7. Draw the sphere's frame
// 8. The principle, as I understand it is that all the lines should
// find themselves matched to a "1" in the stencil buffer and therefore
// be ignored for rendering. Only lines on the edges of the model should
// have half their width not failing the stencil test.
glStencilFunc(GL_EQUAL, 0x0, 0x1);
//glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glLineWidth(3);
glPolygonMode(GL_FRONT_AND_BACK, GL_LINE);
psa::shader::setUniform("outlining", 1);
sphe->show();
psa::shader::setUniform("outlining", 0);
Now just to prove a point, I tired to do something different using the stencil buffer - I just wanted to make sure that everything was in place in my code, for it to work.
** Again I can unfortunately not show a screen capture of the result I get: the scene is like this
http://mathworld.wolfram.com/images/eps-gif/SphereSphereInterGraphic_700.gif
But the smaller sphere is invisible (RGB mask deactivated) and one can see the world background through the hole (instead of the inside of the bigger sphere - face culling is deactivated).
And this is the code... Interestingly, I can change many things like activate/deactivate the STENCIL_TEST, change operation to GL_KEEP everywhere, or even change second stencilFunc to "NOT EQUAL 0"... The result is always the same! I think I am missing something basic here.
void testStencil()
{
// 1. Write a 1 in the Stencil buffer for
// every pixels of the first sphere:
// All colors disabled, we don't need to see that sphere
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE); // Edit: added this
{
sphe->W = mat4::trans(psa::vec4(1.0, 1.0, 1.0)) * mat4::scale(0.9);
sphe->show();
}
// 2. Draw the second sphere with the following rule:
// fail the stencil test for every pixels with a 1.
// This means that any pixel from first sphere will
// not be draw as part of the second sphere.
glStencilFunc(GL_EQUAL, 0x0, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE); // Edit: added this
{
sphe->W = mat4::trans(psa::vec4(1.2, 1.2, 1.2)) * mat4::scale(1.1);
sphe->show();
}
}
Et voila! If anyone could point me in the right direction I would very much appreciate it. I'll also make sure to refer your answer to this other post I found.
