0

I'm creating a graphics application using DirectX9 using DXUT. In my application I would like to implement deferred shading. This method of lighting requires that I render a stencil volume( mask ) for each light within the scene. To avoid rendering the volumes individually I would like to use DirectX's instancing capablilty. Using vertex shader 3.0 and pixel shader 3.0 I have been successful in drawing multiple instances of any given model, however, it seems that when I try to draw the instanced stencil volumes, no data is written to the depth stencil surface of the backbuffer.

My implementation of deferred shading is as follows in pseudocode:

Draw All Scene Geometry( Instanced Geometry Pass ) to the Backbuffer:

  • Enable Z( D3DRS_ZENABLE, TRUE )
  • Enable ZWrites( D3DRS_ZWRITEENABLE, TRUE )
  • Disable Stencil( D3DRS_STENCILENABLE, FALSE )
  • Set CullMode( D3DRS_CULLMODE, D3DCULL_CCW )
  • Disable ColorWrites( D3DRS_COLORWRITEENABLE, 0 )
  • Disable Alphablending( D3DRS_ALPHABLENDENABLE, FALSE )
  • RenderScene

Get a pointer to the backbuffer:

  • IDirect3DDevice9::GetRenderTarget( 0, pBackbufferSurface )

Set 3 rendertargets( COLOR( 0 ), NORMAL( 1 ), POSITION( 2 ) ):

  • IDirect3DDevice9::SetRenderTarget( 0 -> 2, COLOR -> POSITION )

Draw All Scene Geometry to the 3 Render Targets:

  • Enable ColorWrites( D3DRS_COLORWRITEENABLE, CW_RED, CW_GREEN, CW_BLUE, CW_ALPHA )
  • Render Scene

Reset the backbuffer to device RenderTarget 0:

  • IDirect3DDevice9::SetRenderTarget( 0, pBackbufferSurface )

Set the effect texture( SrcColor ) to the COLOR render target

  • ID3DXEffect::SetTexture( "g_TextureSrcColor", pRTColor )

Render a fullscreen quad and sample from the COLOR render target( paints the scene )

Stencil All Scene Lights using Instancing Technique( here is actual code ):

// set the instance buffer vertex declaration
    hr = pd3dDevice->SetVertexDeclaration( CContentManager::GetInstanceBufferVertexDeclaration() );
    if( FAILED( hr ) )
    {
        DebugStringDX( ClassName, "Failed to IDirect3DDevice9::SetVertexDeclaration( instances )at RenderScene()", __LINE__, hr );
        return hr;
    }

    // render instances of each light( sphere and cone )
    for( UINT i = 0; i < 2; ++i )
    {
        // point to base model
        CXModel *pXBase = ( CXModel* )pCM->GetAppObject( i );
        if( !pXBase )
        {
            DebugStringDX( ClassName, 
                "Failed to retrieve the base light model at index( " + ToString( i ) + " ) at Render()", 
                __LINE__,
                E_POINTER );
            continue;
        }

        // if the model has no instances
        //      there is no reason to try and draw
        UINT ucInstances = pXBase->GetNumInstances();
        if( ucInstances < 1 )
            continue;

        // Set up the geometry data stream
        hr = pd3dDevice->SetStreamSourceFreq( 0,
            ( D3DSTREAMSOURCE_INDEXEDDATA | ucInstances ) );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to IDirect3DDevice9::SetStreamSourceFreq( source )( index - " + ToString( i ) + " ) at RenderScene()", __LINE__, hr );
            continue;
        }

        // Set up the instance data stream
        IDirect3DVertexBuffer9 *pVBInstances = pXBase->GetInstanceBuffer();
        hr = pd3dDevice->SetStreamSource( 1, pVBInstances, 0, 
            sizeof( CXModel::sInstanceEntry ) );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to IDirect3DDevice9::SetStreamSource( instances )( index - " + ToString( i ) + " ) at RenderScene()", __LINE__, hr );
            continue;
        }

        hr = pd3dDevice->SetStreamSourceFreq( 1,
            ( D3DSTREAMSOURCE_INSTANCEDATA | 1ul ) );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to IDirect3DDevice9::SetStreamSourceFreq( instances )( index - " + ToString( i ) + " ) at Render()", __LINE__, hr );
            continue;
        }

        // get light model frame root
        pFrame = pXBase->GetFrameRoot();    
        if( !pFrame )
        {
            DebugStringDX( ClassName, "Failed to CXModel::GetFrameRoot( Light Model ) at Render()", __LINE__, hr );
            continue;
        }

        //------------------------------------------------------------------------------------------------------------
        // Stencil Volume Mask Pass
        //------------------------------------------------------------------------------------------------------------
        dev.SetRenderState( D3DRS_ZENABLE, TRUE );
        dev.SetRenderState( D3DRS_ZWRITEENABLE, FALSE );
        dev.SetRenderState( D3DRS_STENCILENABLE, TRUE );
        dev.SetRenderState( D3DRS_STENCILFUNC, D3DCMP_ALWAYS );
        dev.SetRenderState( D3DRS_STENCILFAIL, D3DSTENCILOP_KEEP );
        dev.SetRenderState( D3DRS_STENCILZFAIL, D3DSTENCILOP_INCR );
        dev.SetRenderState( D3DRS_STENCILPASS, D3DSTENCILOP_KEEP );
        dev.SetRenderState( D3DRS_STENCILWRITEMASK, 0xffffffff );
        dev.SetRenderState( D3DRS_CULLMODE, D3DCULL_NONE );
        dev.SetRenderState( D3DRS_COLORWRITEENABLE, 0 );

        hr = BeginPass( CFXDeferred::STENCIL );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to BeginPass( StencilVolumeMask ) at Render()", __LINE__, hr );
            continue;
        }

        // draw the light model
        hr = DrawStaticModel( pd3dDevice, pFrame, &mWorld, pmView, pmProjection, true, bRenderStrips );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to DrawStaticModel( Light Model ) at Render()", __LINE__, hr );
            continue;
        }

        hr = pE->EndPass();
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to ID3DXEffect::EndPass() at Render()", __LINE__, hr );
            continue;
        }

        //------------------------------------------------------------------------------------------------------------
        // Diffuse Light Pass
        //------------------------------------------------------------------------------------------------------------
        dev.SetRenderState( D3DRS_ALPHABLENDENABLE, TRUE );
        dev.SetRenderState( D3DRS_SRCBLEND, D3DBLEND_ONE );
        dev.SetRenderState( D3DRS_DESTBLEND, D3DBLEND_ONE );
        dev.SetRenderState( D3DRS_ZENABLE, FALSE );
        dev.SetRenderState( D3DRS_CULLMODE, D3DCULL_CW );
        dev.SetRenderState( D3DRS_COLORWRITEENABLE, 
            D3DCOLORWRITEENABLE_RED | D3DCOLORWRITEENABLE_GREEN | D3DCOLORWRITEENABLE_BLUE | D3DCOLORWRITEENABLE_ALPHA );
        dev.SetRenderState( D3DRS_STENCILENABLE, TRUE );
        dev.SetRenderState( D3DRS_STENCILFUNC, D3DCMP_EQUAL );
        dev.SetRenderState( D3DRS_STENCILMASK, 0x1 );
        dev.SetRenderState( D3DRS_STENCILPASS, D3DSTENCILOP_ZERO );
        dev.SetRenderState( D3DRS_STENCILREF, 0x1 );
        dev.SetRenderState( D3DRS_STENCILZFAIL, D3DSTENCILOP_KEEP );
        dev.SetRenderState( D3DRS_STENCILFAIL, D3DSTENCILOP_ZERO );

        hr = BeginPass( CFXDeferred::DIFFUSELIGHTSTENCIL );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to BeginPass( DiffuseSpecularLightStencilPass ) at Render()", __LINE__, hr );
            continue;
        }

        // draw light model
        hr = DrawStaticModel( pd3dDevice, pFrame, &mWorld, pmView, pmProjection, true, bRenderStrips );
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to DrawStaticModel( Light Model ) at Render()", __LINE__, hr );
            continue;
        }

        hr = pE->EndPass();
        if( FAILED( hr ) )
        {
            DebugStringDX( ClassName, "Failed to ID3DXEffect::EndPass() at Render()", __LINE__, hr );
            continue;
        }
    }

Here is the instance buffer vertex declaration:

// create instance vertex declaration
    D3DVERTEXELEMENT9 pElements[] = {
        { 0, 0, D3DDECLTYPE_FLOAT3, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_POSITION, 0 },
        { 0, 12, D3DDECLTYPE_FLOAT3, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_NORMAL, 0 },
        { 0, 24, D3DDECLTYPE_FLOAT2, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 0 },
        { 1, 0, D3DDECLTYPE_FLOAT4, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 1 },
        { 1, 16, D3DDECLTYPE_FLOAT4, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 2 },
        { 1, 32, D3DDECLTYPE_FLOAT4, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 3 },
        { 1, 48, D3DDECLTYPE_FLOAT4, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 4 },
        { 1, 64, D3DDECLTYPE_FLOAT2, D3DDECLMETHOD_DEFAULT, D3DDECLUSAGE_TEXCOORD, 5 },
        D3DDECL_END()
    };

This method works fine without instancing...Does instancing prevent depth stencil writes? When I use PIX to debug, it indicates that the "Pixel failed the stencil test". Here is the call to clear the backbuffer before rendering:

pd3dDevice->Clear( 0, NULL, D3DCLEAR_TARGET | D3DCLEAR_ZBUFFER | D3DCLEAR_STENCIL, D3DCOLOR_ARGB( 0, 0, 0, 0 ), 1.0f, 0 ) );

During the creation of the device, I modify the settings to use AutoDepthStencilFormat D3DFMT_D24S8 like so:

// check for D3DFMT_D24S8 AutoDepthStencilFormat
    // if present set the AutoDepthStencilFormat to D3DFMT_D24S8
    if( DXUTGetD3D9Enumeration()->GetPossibleDepthStencilFormatList()->Contains( D3DFMT_D24S8 ) )
    {
        pDeviceSettings->d3d9.pp.AutoDepthStencilFormat = D3DFMT_D24S8;
    }

If I change the stencil parameter( the last parameter ) of IDirect3DDevice9::Clear() to 1. The stencil volume passes the stencil test...this is because I set the stencil operation to increase the stencil value by 1 if a pixel of the volume mask intersects with a pixel on the backbuffer, and on that event, admit the light's pixel color. This indicates that there is one of two problems:

  1. When drawing the scene( objects ) during the 'Instanced Geometry Pass'...the backbuffer is not storing the depth of the objects as they are drawn( so the stencil test has a '0' value for comparison or
  2. When drawing the stencil volumes( lights ) the backbuffer is not storing the depth of the volumes as they are drawn...

Is this due to my instancing method? Is it due to the Shader Versions( vs3, ps3 ) or the version of DX?

P. Avery
  • 779
  • 1
  • 16
  • 34
  • "*This method of lighting requires that I render a stencil volume( mask ) for each light within the scene.*" Generally speaking, deferred rendering and *stencil shadows* don't work together. Like, *at all*. You need to switch to shadow maps. – Nicol Bolas Jul 25 '13 at 20:49
  • who said anything about shadows? no shadows...i'm implementing a deferred lighting technique that uses stencil volumes ( post-processing ). All pixels that pass the stecil test and fail the depth test are "lit" for a given volume( representation of a light - point uses sphere, spot uses cone )... – P. Avery Jul 25 '13 at 21:39
  • Then what is the "stencil volume" a stencil volume of? If it's not a shadow, then what is the stenciling supposed to do in your scene? – Nicol Bolas Jul 25 '13 at 21:42
  • each stencil volume is either a sphere or a cone...all pixels that pass the stencil test and fail the depth test are intersecting with the stencil volume. I can then add the color from the light( represented in form by its stencil volume ) for all pixels that qualified... – P. Avery Jul 25 '13 at 21:47

1 Answers1

0

The problem was that I compiled the vertex shader( for stencil volume mask ) with vs_3_0...I changed it to vs_2_0 and now it works fine...I don't understand because I thought it was necessary to use vertex shader model 3.0 with instancing...

P. Avery
  • 779
  • 1
  • 16
  • 34