Outlining an object using a stencil buffer give wrong result

Question

I'm trying to use the stencil buffer to draw outline border behind a model.

I'm using 2 render passes, first rendering the model and using GL_ALWAYS to write 1 to the stencil buffer and then I render a slightly scaled up version of the model that uses GL_NOTEQUAL with ref value of 1 so basically to pass only fragments that are outside the original cube thus making the highlight.

I'm getting weird result, the outline is not fully rendered and when I move my camera enough it disappears completely as if the stencil test of the scaled up cube doesn't pass at all. Here's the gist of the rendering loop (shader3 just outputs a color for the border):

while (!glfwWindowShouldClose(window))
{
    glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
    glClearStencil(0);
    renderer.Clear();    
    ...
    glm::mat4 projection = glm::perspective(glm::radians(45.0f), (float)SCREEN_WIDTH / (float)SCREEN_HEIGHT, 0.1f, 100.0f);
    UniformMat4f projectionUniform("u_Projection");
    projectionUniform.SetValues(projection);

    glm::mat4 view = camera.GetView();
    UniformMat4f viewUniform("u_View");
    viewUniform.SetValues(view);

    modelUniform.SetValues(glm::translate(glm::mat4(1.0f), glm::vec3(5.0f, 0.0f, 0.0f)));

    shader1.SetUniform(projectionUniform);
    shader1.SetUniform(viewUniform);
    shader1.SetUniform(modelUniform);

    glStencilFunc(GL_ALWAYS, 1, 0xFF);
    glStencilMask(0xFF);

    renderer.Draw(model, shader1);

    modelUniform.SetValues(glm::scale(glm::translate(glm::mat4(1.0f), glm::vec3(5.0f, 0.0f, 0.0f)), glm::vec3(1.1f, 1.1f, 1.1f)));
    shader3.SetUniform(projectionUniform);
    shader3.SetUniform(viewUniform);
    shader3.SetUniform(modelUniform);

    glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
    glStencilMask(0x00);
    glDisable(GL_DEPTH_TEST);
    renderer.Draw(model, shader3);
    glEnable(GL_DEPTH_TEST);
}

I first tried it with the nanosuit model from crysis and it work as I described not showing all the parts and eventually disappearing when moving the camera. I've decided to try a simple geometry to test this out and tried to outline a simple cube and got the same result:

Answer 1

The issue is the call to glStencilMask(0x00); at the end of the main loop. the default value of the stencil mask is 0xff (for 8 bit, see glStencilMask). OpenGL is a state engine. A state is kept, till it is changed again (even beyond frames).
glClear considers the state of the color, depth and stencil mask. If the stencil mask is 0x00, then the stencil buffer isn't cleared at all.

Set the stencil mask 0xff, before you clear the stencil buffer to solve the issue:

glStencilMask(0xFF);
renderer.Clear();