opengl stencils seem to have no effect

Question

I am attempting to create slightly-rounded rectangles - the idea is that these would be platforms in a game - by creating a solid white rectangle, outlining it with black rectangles, and putting circles in the corners. To keep things simple, my problem is essentially the same as this: How to set blend function on opengl for two overlapping objects

I want to tackle this using stencil testing, since I've looked it up and it seems to be a good way to do it.

This is what the render function of a platform object looks like:

    GL11.glClear(GL11.GL_STENCIL_BUFFER_BIT);

    // Setup stencil such that stencil = 0x01 isn't drawn over
    GL11.glEnable(GL11.GL_STENCIL_TEST);
    GL11.glStencilFunc(GL11.GL_ALWAYS, 1, 0xFF); // Do not draw if stencil byte = 1
    GL11.glStencilOp(GL11.GL_KEEP, GL11.GL_KEEP, GL11.GL_REPLACE);
    // stencil is on (mask is all 1s)
    GL11.glStencilMask(0xFF);
    inside.render(st);

    GL11.glStencilFunc(GL11.GL_EQUAL, 1, 0xFF); // Do not draw if stencil byte = 1
    left.render(st);
    bottom.render(st);
    top.render(st);
    right.render(st);

    GL11.glDisable(GL11.GL_STENCIL_TEST);

inside, left, bottom, top, and right are all objects of my own Polygon class, which basically just draws the triangles for a given polygon (actually, right now it can only draw rectangles I just realized... but I digress)

As far as I understand it, this is how my program should be working:

1. Stencil test is turned on, then my first calls to glStencilFunc and glStencilOp make anything drawn also change that area on the stencil buffer to 1.
2. Inside rectangle for the platform is drawn. This area should be all 1s on the stencil buffer now.
3. A call to glStencilFunc now prevents drawing over areas with a stencil value of 1
4. The borders of the platform are drawn. Technically these rectangles overlap the inner-rectangle half way into the border's width, but since the inner-rectangle is all 1s on the stencil buffer nothing should be drawn there.
5. Eventually, I'll have circles rendered on the corners. (which is why drawing the border lines should also change those areas to all 1s in the stencil buffer).

So, I feel like I've done my research, but it's really not working. Is three a major OpenGL feature I might be missing in order to make stencils work?

Answer 1

To use stencil functionality, your framebuffer must have a stencil buffer. Since most applications are not using a stencil buffer, you will mostly get a framebuffer without stencil by default. To make sure that you have a stencil buffer, you need to explicitly request it.

How a stencil buffer is requested depends on the platform and window system interface. It always needs to be done during initialization, and is part of setting up the context and rendering surface. Typical terminology used in the process of specifying the buffers you need are "pixel format" and "configuration".

From reading the documentation, this is specified as part of the PixelFormat class in LWJGL. For creating the PixelFormat:

PixelFormat pixFmt = new PixelFormat().withStencilBits(8);

This PixelFormat instance can then be passed to one of the create() method overloads of the Display class.