Stencil Options between DirectX and OpenGL

Question

I'm implementing an emulation layer. Everything otherwise works, except the stencil stuff (my shadows overlap).

I'm just wondering if I'm doing things logically - if I'm making the proper conclusions/assumptions:

case D3DRS_STENCILENABLE:
    glAble(GL_STENCIL_TEST, value);
    break;
case D3DRS_STENCILFAIL:
    sStencilFail = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILZFAIL:
    sStencilPassDepthFail = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILPASS:
    sStencilPassDepthPass = glFromD3DSTENCILOP((D3DSTENCILOP)value);
    AssertGL(glStencilOp(sStencilFail, sStencilPassDepthFail, sStencilPassDepthPass));
    break;
case D3DRS_STENCILFUNC:
    sStencilFunc = glFromD3DCMPFUNC((D3DCMPFUNC)value);
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILREF:
    sStencilRef = value;
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILMASK:
    sStencilValueMask = value;
    AssertGL(glStencilFunc(sStencilFunc, sStencilRef, sStencilValueMask));
    break;
case D3DRS_STENCILWRITEMASK:
    AssertGL(glStencilMask(value));
    break;

The following are used above. glAble() simply is a wrapper for glEnable/glDisable.

static GLenum glFromD3DCMPFUNC(D3DCMPFUNC value) {
    return(GL_NEVER + value - 1);
}

static GLenum glFromD3DSTENCILOP(D3DSTENCILOP value) {
    switch (value) {
    case D3DSTENCILOP_KEEP: return(GL_KEEP);
    case D3DSTENCILOP_ZERO: return(GL_ZERO);
    case D3DSTENCILOP_REPLACE: return(GL_REPLACE);
    case D3DSTENCILOP_INVERT: return(GL_INVERT);
    case D3DSTENCILOP_INCRSAT:
    case D3DSTENCILOP_INCR:
        return(GL_INCR);
    case D3DSTENCILOP_DECRSAT:
    case D3DSTENCILOP_DECR:
        return(GL_DECR);
    default: Assert(!"Unsupported!"); return(0);
    }
}

Which versions of D3D and GL are you targeting? You may have trouble implementing D3D9 stenciling in versions of GL older than 2.0, which only expose two-sided stencil operations through extensions. I would expect to see something in your emulator that handles `D3DRS_Two_Sided_StencilMODE` and then dispatches things like `D3DRS_CCW_STENCILFAIL` to `glStencilFuncSeparate (...)` appropriately. It adds quite a bit of complication, but there are common use-cases for handling the front/back stencil test differently. — Andon M. Coleman, Oct 14 '13 at 17:01

score 1 · Answer 1 · answered Oct 15 '13 at 04:17

There are a couple of things that could go wrong.

You do the same for incr and incr_sat. Is the test code using it? Gl needs an extension to saturate. It's not the same thing.
With old d3d code double check that the stencil ref is 0..255. There might be a difference how a parameter >255 or <0 is handled (clamp vs. and)
Double check that d3d is not doing two sided. And remember that the culling is flipped.
Double check the stencil clear code also!

The good news is, that this should work with basic table translation. There is no fundamental API issue (I did it before). It just needs a lot of testing because the parameter space is large.

My code does not use saturation. And my values are either 0 or 1 (so it's not an issue of clamping). I have D3D as not doing two-sided and that the culling is different - I see the shadows but they're overlapping. The stencil is definitely being cleared, if it wasn't, there would be no shadows to begin with. Are there any other things I could watch out for? — Mike Weir, Oct 15 '13 at 12:53

score 1 · Accepted Answer · answered Oct 15 '13 at 17:29

1

Figured it out, I forgot to set my PIXELFORMATDESCRIPTOR:

pfd.cStencilBits = 8;

answered Oct 15 '13 at 17:29

Mike Weir

3,094
1
30
46

Stencil Options between DirectX and OpenGL

2 Answers2