DirectX alpha masking

Question

I'm working on a game using DirectX 9. Here's what I'm trying to do:

After the scene is rendered, on top of it I want to render few sprites: a black cover on entire scene and a few sprites, which are masks showing where the cover should have holes. So far I tried messing with blend mode but with no luck. My code setting it up looks like this:

D3DD->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);
D3DD->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
D3DD->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);

I guess the best way would be to multiply each sprites alpha, but according to http://msdn.microsoft.com/en-us/library/windows/desktop/bb172508%28v=vs.85%29.aspx no such mode is supported. Is There another way to do this?

edit

Following Nico Schertler's answer, here's the code I came up with:

LPDIRECT3DTEXTURE9      pRenderTexture;
LPDIRECT3DSURFACE9      pRenderSurface,
                        pBackBuffer;

// create texture
D3DD->CreateTexture(1024,
                    1024,
                    1,
                    D3DUSAGE_RENDERTARGET,
                    D3DFMT_R5G6B5,
                    D3DPOOL_DEFAULT,
                    &pRenderTexture,
                    NULL);

pRenderTexture->GetSurfaceLevel(0,&pRenderSurface);

// store old render target - back buffer
D3DD->GetRenderTarget(0,&pBackBuffer);

// set new render target - texture
D3DD->SetRenderTarget(0,pRenderSurface);

//clear texture to opaque black
D3DD->Clear(0,
            NULL,
            D3DCLEAR_TARGET | D3DCLEAR_ZBUFFER,
            D3DCOLOR_XRGB(0,0,0),
            32.0f,
            0);

// set blending
D3DD->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_ZERO);
D3DD->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_ONE);
D3DD->SetRenderState(D3DRS_SRCBLENDALPHA, D3DBLEND_ZERO);
D3DD->SetRenderState(D3DRS_DESTBLENDALPHA, D3DBLEND_SRCALPHA);
D3DD->SetRenderState(D3DRS_SEPARATEALPHABLENDENABLE, TRUE);

//// now I render hole sprites the usual way

// restore back buffe as render target
D3DD->SetRenderTarget(0,pBackBuffer);

// restore blending
D3DD->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
D3DD->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
D3DD->SetRenderState(D3DRS_SRCBLENDALPHA, D3DBLEND_SRCALPHA);
D3DD->SetRenderState(D3DRS_DESTBLENDALPHA, D3DBLEND_INVSRCALPHA);
D3DD->SetRenderState(D3DRS_SEPARATEALPHABLENDENABLE, FALSE);

ulong color = ulong(-1);
Vertex2D v[4];

v[0] = Vertex2D(0, 0, 0);
v[1] = Vertex2D(1023, 0, 0);
v[3] = Vertex2D(1023, 1023, 0);
v[2] = Vertex2D(0, 1023, 0);

D3DD->SetTexture(0, pRenderTexture);
D3DD->SetFVF(D3DFVF_XYZRHW | D3DFVF_DIFFUSE | D3DFVF_TEX1);
D3DD->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, v, sizeof(Vertex2D));

D3DD->SetTexture(0, NULL);

// release used resources
pRenderTexture->Release();
pRenderSurface->Release();
pBackBuffer->Release();

Unfortunatelly, the app crashes when restoring the old render target. Any advice?

What error do you get? Btw, you should use a texture format with an alpha channel, of course. — Nico Schertler, Jul 31 '13 at 10:26
Access violation reading location 0xcdcdcdcd. The pBackBuffer pointer seem to be OK up to the point where I render the sprites, then in D3DD->SetRenderTarget(0,pBackBuffer) its value is different (always 0xcdcdcdcd). — mag_zbc, Jul 31 '13 at 10:34
That's strange. Do you do anything with the pointer in between? Can you locate the line where the pointer is changed? — Nico Schertler, Jul 31 '13 at 10:36
Found the cause, of course it was a stupid mistake. Nothing crashes anymore, but it doesn't seem to be working - the black cover is not visible. — mag_zbc, Jul 31 '13 at 10:41
Then I can just advice to debug the problem step by step. Firstly, remove the rendering of the hole sprites. Then the texture must be black. If not, that's the problem. After that, switch the blend states to some states you know and verify the result. Then step by step reactivate the steps and see where the problem is. — Nico Schertler, Jul 31 '13 at 10:48
Apparently something's wrong with the way I create the texture - when I use already existing texture, which is a black sprite, the cover is rendered with a hole just like it's supposed to, but it stays in the same place it was rendered for the first time - I cant move it around. — mag_zbc, Jul 31 '13 at 11:20
Are you sure that clearing the z-buffer to 32 is what you want? Have you tried to save the texture to a file and examine its contents? — Nico Schertler, Jul 31 '13 at 11:55

Nico Schertler · Answer 1 · 2013-07-30T18:04:07.147

Firstly, you should create the mask in a separate texture first. Then you can add the holes as needed. Finally, draw the mask on the screen:

Initialize the texture
Clear it to opaque black
Using the following blend states:
    D3DRS_SRCBLEND -> D3DBLEND_ZERO (hole's color does not matter)
    D3DRS_DESTBLEND -> D3DBLEND_ONE (preserve the black color)
    D3DRS_SRCBLENDALPHA -> D3DBLEND_ZERO
    D3DRS_DESTBLENDALPHA -> D3DBLEND_SRCALPHA
    D3DRS_SEPARATEALPHABLENDENABLE -> TRUE
Draw each hole sprite
Restore default blending (src_alpha / inv_src_alpha)
Render the texture as a sprite to the back buffer

The above blend state assumes that the holes are opaque where there should be a hole. Then, the color is calculated by:

blended color = 0 * hole sprite color + 1 * background color

which should always be black.

And the alpha channel is calculated by:

blended alpha = 0 * hole sprite alpha + (1 - hole sprite alpha) * background alpha

So where the hole sprite is opaque, the blended alpha becomes 0. Where it is transparent, the blended alpha is the previous value. Values in between are blended.

DirectX alpha masking

1 Answers1